Performance of ext4

Performance of ext4

[Holger Kiehl]

Hello

Doing some performance test between ext3 and ext4 I noticed that ext4
is not much faster or in some cases slower then ext3. Two years ago when
I tested ext4 it was a lot faster then ext3 (see my mail:
http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
I got the following results:

Version  1.03        ------Sequential Output------ --Sequential Input- --Random-
                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine         Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ext3(2 years ago)16G 38621  98 194816 94 87776  49 37921  92 239128 54  1402   5
                  16G 47000  99 194276 94 89232  49 38628  92 240539 55  1399   5
                  16G 45873  98 178195 90 89726  50 38482  92 240490 55  1381   4

ext3 (now)       16G 51501  97 210601 91 100479 32 55528  98 301589 44  1198   5
                  16G 52702  98 215540 94 99339  32 55376  97 300933 44  1159   4
                  16G 52426  99 212584 94 99091  31 55656  98 301669 44  1160   4

ext4(2 years ago)16G 59223  91 264155 45 111459 36 57313  99 317944 63  1478   7
                  16G 58814  92 276803 47 110418 36 57105  99 317534 65  1525   5
                  16G 58299  92 274523 48 110290 36 56723  99 318839 65  1502   4

ext4 (now)       16G 52965  98 224199 89 108440 32 56389  99 303792 42  1526   4
                  16G 52792  98 223980 92 107685 32 56350  98 303066 42  1532   4
                  16G 52994  98 222354 92 107802 32 56386  99 303727 41  1455   4

For this system the write performance is the most important factor and one
can see today ext4 is marginally faster then ext3. But 2 years ago ext4 was
a lot faster (~270MB against ~223MB).

Using my own benchmark afdbench where many process copy thousands of small
files around the results are as follows:

   For ext3: 5449.76 files per second 15.58 MiB/s
   For ext4: 5162.16 files per second 15.48 MiB/s

So in this test ext4 is a bit slower then ext3. Since afdbench has seen
considerable changes, one cannot compare these results with those 2 years
ago. But 2 years ago ext4 was 12% faster then ext3.

Test where done with kernel 2.6.25.4 and file system where created as follows:

    ext3: mke2fs -b 4096 -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7
    ext4: mke2fs -b 4096 -E test_fs -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7

And both where mounted with the following options:

    noatime,nodiratime,commit=15

2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
happened with the performance of ext4? I noticed that 2 years ago I could
use extents+mballoc+delalloc, now there is only extents+mballoc in the
current kernels. Could delalloc make the big difference? I saw that
in Andrew Morton mm tree delalloc is included. Unfortunately when I tried using
2.6.26-rc2-mm1 a sync would never return and there where lot of other
odd things, so I could not do any tests with delalloc.

So any idea what I am doing wrong or what I could do to improve those numbers?
Please CC me since I am not subscribed to the list.

Thanks,
Holger

Re: Performance of ext4

[Aneesh Kumar K.V]

On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
> Hello
>
> Doing some performance test between ext3 and ext4 I noticed that ext4
> is not much faster or in some cases slower then ext3. Two years ago when
> I tested ext4 it was a lot faster then ext3 (see my mail:
> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
> I got the following results:
>
> Version  1.03        ------Sequential Output------ --Sequential Input- --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine         Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> ext3(2 years ago)16G 38621  98 194816 94 87776  49 37921  92 239128 54  1402   5
>                  16G 47000  99 194276 94 89232  49 38628  92 240539 55  1399   5
>                  16G 45873  98 178195 90 89726  50 38482  92 240490 55  1381   4
>
> ext3 (now)       16G 51501  97 210601 91 100479 32 55528  98 301589 44  1198   5
>                  16G 52702  98 215540 94 99339  32 55376  97 300933 44  1159   4
>                  16G 52426  99 212584 94 99091  31 55656  98 301669 44  1160   4
>
> ext4(2 years ago)16G 59223  91 264155 45 111459 36 57313  99 317944 63  1478   7
>                  16G 58814  92 276803 47 110418 36 57105  99 317534 65  1525   5
>                  16G 58299  92 274523 48 110290 36 56723  99 318839 65  1502   4
>
> ext4 (now)       16G 52965  98 224199 89 108440 32 56389  99 303792 42  1526   4
>                  16G 52792  98 223980 92 107685 32 56350  98 303066 42  1532   4
>                  16G 52994  98 222354 92 107802 32 56386  99 303727 41  1455   4
>
> For this system the write performance is the most important factor and one
> can see today ext4 is marginally faster then ext3. But 2 years ago ext4 was
> a lot faster (~270MB against ~223MB).
>
> Using my own benchmark afdbench where many process copy thousands of small
> files around the results are as follows:
>
>   For ext3: 5449.76 files per second 15.58 MiB/s
>   For ext4: 5162.16 files per second 15.48 MiB/s
>
> So in this test ext4 is a bit slower then ext3. Since afdbench has seen
> considerable changes, one cannot compare these results with those 2 years
> ago. But 2 years ago ext4 was 12% faster then ext3.
>
> Test where done with kernel 2.6.25.4 and file system where created as follows:
>
>    ext3: mke2fs -b 4096 -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7
>    ext4: mke2fs -b 4096 -E test_fs -m 0 -O dir_index,large_file,filetype,has_journal,sparse_super -j /dev/md7
>
> And both where mounted with the following options:
>
>    noatime,nodiratime,commit=15
>
> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
> happened with the performance of ext4? I noticed that 2 years ago I could
> use extents+mballoc+delalloc, now there is only extents+mballoc in the
> current kernels. Could delalloc make the big difference? I saw that
> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried using
> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
> odd things, so I could not do any tests with delalloc.


The sync and other related hangs should be fix with the latest patches
vailable at http://repo.or.cz/w/ext4-patch-queue.git. Using mballoc have
an impact on CPU utilization because we try to build an in-memory extent
map of free blocks available in the group. The cold cache run (the first
run) would take more time because of the time needed to build the extent
map. So repeating the same test and looking at the numbers would help us
understand the impact of in-memory extent building code.

>
> So any idea what I am doing wrong or what I could do to improve those numbers?
> Please CC me since I am not subscribed to the list.
>

You should be able to apply the patches in the patchqueue mentioned above  to 2.6.26-rc5
Can you test with the same and get the numbers. ? Also delalloc enabled
by default with changes from patchqueue force writeback mode for
journal. So you may want to enable writeback for ext3.

-aneesh

Re: Performance of ext4

[Holger Kiehl]

On Wed, 11 Jun 2008, Aneesh Kumar K.V wrote:

> On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>>
>> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
>> happened with the performance of ext4? I noticed that 2 years ago I could
>> use extents+mballoc+delalloc, now there is only extents+mballoc in the
>> current kernels. Could delalloc make the big difference? I saw that
>> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried using
>> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
>> odd things, so I could not do any tests with delalloc.
>
>
> The sync and other related hangs should be fix with the latest patches
> vailable at http://repo.or.cz/w/ext4-patch-queue.git. Using mballoc have
> an impact on CPU utilization because we try to build an in-memory extent
> map of free blocks available in the group. The cold cache run (the first
> run) would take more time because of the time needed to build the extent
> map. So repeating the same test and looking at the numbers would help us
> understand the impact of in-memory extent building code.
>
Not sure if I understand this. I first formated the file system and run the
test three times.

>>
>> So any idea what I am doing wrong or what I could do to improve those numbers?
>> Please CC me since I am not subscribed to the list.
>>
>
> You should be able to apply the patches in the patchqueue mentioned above  to 2.6.26-rc5
> Can you test with the same and get the numbers. ? Also delalloc enabled
> by default with changes from patchqueue force writeback mode for
> journal. So you may want to enable writeback for ext3.
>
Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
Or did you mean to apply those against the mm tree?

Thanks,
Holger

Re: Performance of ext4

[Nick Dokos]

Holger Kiehl <Holger.Kiehl@dwd.de> wrote:

> Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
> The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
> Or did you mean to apply those against the mm tree?
> 

Just an FYI and a data point: I was able to apply the patch queue
against 2.6.26-rc5 (after I set up quilt correctly) without any
problems.

Regards,
Nick

Re: Performance of ext4

[Holger Kiehl]

On Wed, 11 Jun 2008, Nick Dokos wrote:

> Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
>
>> Against a clean 2.6.26-rc5 the patchqueue does not apply. Lots of rejects.
>> The patchqueue also contains changes for defrag that are not in 2.6.26-rc5.
>> Or did you mean to apply those against the mm tree?
>>
>
> Just an FYI and a data point: I was able to apply the patch queue
> against 2.6.26-rc5 (after I set up quilt correctly) without any
> problems.
>
I have never used quilt, so how do I set it up correctly? I am using
Fedora 9 and have copied all patches from ext4-patch-queue into
2.6.26-rc5/patches. Then I always issue the following command in
2.6.26-rc5 directory:

      quilt push

That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
it gets lots of rejects. So how do I set this up correctly?

Thanks,
Holger

Re: Performance of ext4

[Solofo.Ramangalahy]

Hi Holger,

Holger Kiehl writes:
 > That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
 > it gets lots of rejects. So how do I set this up correctly?

I think your quilt is set up correctly.

The rejects probably come from the patch queue being based on a
different version than 2.6.26-rc5.

Which patchset did you use?

What I do when wanting a patch queue for a specific kernel is to
search for commit message indicating the rebase for this version.

There is a "Rebase to linux-2.6.26-rc5" on
http://repo.or.cz/w/ext4-patch-queue.git
So the corresponding snapshot
http://repo.or.cz/w/ext4-patch-queue.git?a=snapshot;h=ed6aaac3b0cc0ba47dbb2af0884306584271e5a2;sf=tgz
patches 2.6.26-rc5 smoothly.
(the last patchset also applies, with offset and fuzz, though)

-- 
solofo

Re: Performance of ext4

[Holger Kiehl]

Hello Solofo

On Thu, 12 Jun 2008, Solofo.Ramangalahy@bull.net wrote:

> Hi Holger,
>
> Holger Kiehl writes:
> > That works until it comes to patch patches/ext4_ialloc-flexbg.patch when
> > it gets lots of rejects. So how do I set this up correctly?
>
> I think your quilt is set up correctly.
>
> The rejects probably come from the patch queue being based on a
> different version than 2.6.26-rc5.
>
> Which patchset did you use?
>
> What I do when wanting a patch queue for a specific kernel is to
> search for commit message indicating the rebase for this version.
>
> There is a "Rebase to linux-2.6.26-rc5" on
> http://repo.or.cz/w/ext4-patch-queue.git
> So the corresponding snapshot
> http://repo.or.cz/w/ext4-patch-queue.git?a=snapshot;h=ed6aaac3b0cc0ba47dbb2af0884306584271e5a2;sf=tgz
> patches 2.6.26-rc5 smoothly.
> (the last patchset also applies, with offset and fuzz, though)
>
Yes, that is the one I took. Just to make sure, I downloaded it again (and
the linux-2.6.26-rc5 tree) and now it works. When I compare the two patchsets
the one I pulled this morning had patches from 7 June and this one has more
and they are from 11th June.

Thanks a lot for your help, at last I now have all the patches and can go
on with testing.

Thanks,
Holger

Re: Performance of ext4

[Theodore Tso]

On Thu, Jun 12, 2008 at 12:00:54PM +0000, Holger Kiehl wrote: 
> Yes, that is the one I took. Just to make sure, I downloaded it
> again (and the linux-2.6.26-rc5 tree) and now it works. When I
> compare the two patchsets the one I pulled this morning had patches
> from 7 June and this one has more and they are from 11th June.

Note that you don't have to download things from scratch, necessarily;
the git command "git pull" will (if you havent made any changes in the
git repository; if you have there are a few more steps you might have
to take) pull down any newer changes and update your patch directory
directly.

Note that after you type "git pull", you may find that the
ext4-patch-queue has been rebased to a newer version of the kernel.
For example, the patch queue is currently against 2.6.26-rc5.  The
latest bleedig edge kernel as released by Linus in his git tree of the
kernel already has these patches:

ext4-fix-use-of-uninitalized-data.patch
ext4-fix-uninit-block-group-initialization-with-FLEX_BG.patch
ext4-fix-onine-resize-bug.patch
fix-journal-checksum-mem-leak
jbd2-if-a-journal-checksum-error-is-detected-propa.patch 
show-journal-async-commit-option
jbd2-Fix-barrier-fallback-code-to-re-lock-the-buffe.patch
ext4-enable-barriers-by-default.patch

from the patch series.  But, he hasn't released an official kernel
release for 2.6.26-rc5 yet.  So in the near future, the ext4 patch
queue will probably get rebased against one of the "daily snapshots",
as found in /pub/linux/kernel/v2.6/snapshots on ftp.kernel.org.  So
for example, 2.6.26-rc5-git6 is the sixth snapshot since -rc5 was
released.  There is a patch against -rc5 named
patch-2.6.26-rc5-git6.bz2 as located in the aforementioned directory
on ftp.kernel.org, or if you are tracking Linus's kernel git tree, the
file patch-2.6.26-rc5-git6.id proclaims that -rc5-git6 is git id
631025b.  You will see that when you look at the first line of the
series file in the ext4 patch queue, where it currently states:

# BASE 2.6.26-rc5

and where it will in the future say something like this:

# BASE 2.6.26-rc5-git6

That can be used by automated scripts to automatically determine which
version of the Linux kernel should be grabbed before trying to apply
the latest version of the ext4 patch queue.  In general we try to use
mainly official -rc1, -rc2, etc. release points, but after Linus
pulled down some stable bug fixes, we will sometimes use a daily git
snapshot release such as -rc5-git6 as described above.

Hope this helps,

							- Ted

Re: Performance of ext4

[Holger Kiehl]

Hello Ted

On Thu, 12 Jun 2008, Theodore Tso wrote:

> On Thu, Jun 12, 2008 at 12:00:54PM +0000, Holger Kiehl wrote:
>> Yes, that is the one I took. Just to make sure, I downloaded it
>> again (and the linux-2.6.26-rc5 tree) and now it works. When I
>> compare the two patchsets the one I pulled this morning had patches
>> from 7 June and this one has more and they are from 11th June.
>
> Note that you don't have to download things from scratch, necessarily;
> the git command "git pull" will (if you havent made any changes in the
> git repository; if you have there are a few more steps you might have
> to take) pull down any newer changes and update your patch directory
> directly.
>
> Note that after you type "git pull", you may find that the
> ext4-patch-queue has been rebased to a newer version of the kernel.
> For example, the patch queue is currently against 2.6.26-rc5.  The
> latest bleedig edge kernel as released by Linus in his git tree of the
> kernel already has these patches:
>
> ext4-fix-use-of-uninitalized-data.patch
> ext4-fix-uninit-block-group-initialization-with-FLEX_BG.patch
> ext4-fix-onine-resize-bug.patch
> fix-journal-checksum-mem-leak
> jbd2-if-a-journal-checksum-error-is-detected-propa.patch
> show-journal-async-commit-option
> jbd2-Fix-barrier-fallback-code-to-re-lock-the-buffe.patch
> ext4-enable-barriers-by-default.patch
>
> from the patch series.  But, he hasn't released an official kernel
> release for 2.6.26-rc5 yet.  So in the near future, the ext4 patch
>
I think this should be 2.6.26-rc6?

> queue will probably get rebased against one of the "daily snapshots",
> as found in /pub/linux/kernel/v2.6/snapshots on ftp.kernel.org.  So
> for example, 2.6.26-rc5-git6 is the sixth snapshot since -rc5 was
> released.  There is a patch against -rc5 named
> patch-2.6.26-rc5-git6.bz2 as located in the aforementioned directory
> on ftp.kernel.org, or if you are tracking Linus's kernel git tree, the
> file patch-2.6.26-rc5-git6.id proclaims that -rc5-git6 is git id
> 631025b.  You will see that when you look at the first line of the
> series file in the ext4 patch queue, where it currently states:
>
> # BASE 2.6.26-rc5
>
> and where it will in the future say something like this:
>
> # BASE 2.6.26-rc5-git6
>
> That can be used by automated scripts to automatically determine which
> version of the Linux kernel should be grabbed before trying to apply
> the latest version of the ext4 patch queue.  In general we try to use
> mainly official -rc1, -rc2, etc. release points, but after Linus
> pulled down some stable bug fixes, we will sometimes use a daily git
> snapshot release such as -rc5-git6 as described above.
>
> Hope this helps,
>
Yes, thanks a lot. I still have not yet managed to get this to work
with git. But downloading linux-2.6.26-rc5.tar.bz2 and then copying
the ext4-patch-queue into linux-2.6.26-rc5/patches and then using
quilt pull did get all the patches applied.

Here are the first test results:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ext4(patchqueue)16G 59727  98 252733 52 110177 25 55821  98 296739 42  1553   5
                 16G 61047  99 239242 48 111664 25 55706  98 297151 42  1545   4
                 16G 60503  99 241532 47 109655 25 55671  98 297648 42  1552   3

That is with 2.6.26-rc5 + ext4-patch-queue and filesystem was created with
latest e2fsprogs snapshot as you suggested and formated with the following
command:

    mke2fs -t ext4dev /dev/md7

Besides, I am doing those tests on a software raid 1+0. I think that is also
the reason why barriers are disabled by defaults:

    Jun 12 12:17:48 helena kernel: kjournald2 starting.  Commit interval 15 seconds
    Jun 12 12:17:48 helena kernel: EXT4 FS on md7, internal journal
    Jun 12 12:17:48 helena kernel: EXT4-fs: mounted filesystem with writeback data mode.
    Jun 12 12:17:48 helena kernel: EXT4-fs: file extents enabled
    Jun 12 12:17:48 helena kernel: EXT4-fs: mballoc enabled
    Jun 12 12:18:26 helena kernel: JBD: barrier-based sync failed on md7 - disabling barriers

If one compares the results with those of 2.6.25.4 (without ext4-patch-queue):

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ext4 (2.6.25.4) 16G 52965  98 224199 89 108440 32 56389  99 303792 42  1526   4
                 16G 52792  98 223980 92 107685 32 56350  98 303066 42  1532   4
                 16G 52994  98 222354 92 107802 32 56386  99 303727 41  1455   4

This is quit an improvement but still does not match those numbers two years
ago.

However there must be still some bug, I am unable to run my own afdbench
benchmark on this (2.6.26-rc5 + ext4-patch-queue). It starts fine but then
about 3 minutes into the test (the test runs ~60 minutes) all process start
hanging in D-state (here a list):

    afdbench 16381  0.0  0.0   4064   760 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd3 0 0 1 22765b95/0/48511fd7_2db_0
    afdbench 16388  0.0  0.0   4064   760 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd3 3 0 1 22765b95/0/48511fd7_2dc_0
    nobody   16391  0.0  0.0  44304  1444 ?        Ss   13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16395  0.0  0.0   4064   760 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd3 5 0 1 22765b95/0/48511fd7_2dd_0
    nobody   16397  0.0  0.0  44304  1448 ?        Ss   13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    nobody   16400  0.0  0.0  44304  1448 ?        Ss   13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16404  0.0  0.0  44328   992 ?        D    13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16409  0.0  0.0  44328   996 ?        D    13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16411  0.0  0.0  44328   996 ?        D    13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16848  0.0  0.0   4064   716 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 2 0 0 820dc3d0/0/48511fdd_2f3_0
    afdbench 16855  0.0  0.0   4064   712 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 1 0 7 cd52409d/0/48511fdd_2f3_0
    afdbench 16857  0.0  0.0   4064   708 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 0 0 8 e1d94fe0/0/48511fdd_2f3_0
    afdbench 16861  0.0  0.0   4064   708 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 2 0 9 26686393/0/48511fdd_2f3_0
    afdbench 16865  0.0  0.0   4064   704 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 0 0 10 ae36cee/0/48511fdd_2f3_0
    afdbench 16871  0.0  0.0   4064   716 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd2 1 0 0 bca9d45f/0/48511fdd_2d8_0
    afdbench 16876  0.0  0.0   4064   716 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 1 0 0 820dc3d0/0/48511fdd_2f4_0
    afdbench 16878  0.0  0.0   4064   772 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 0 0 6 b8cf511a/0/48511fdd_2f4_0
    nobody   16879  0.0  0.0  44304  1448 ?        Ss   13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16881  0.0  0.0   4064   704 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 0 0 7 cd52409d/0/48511fdd_2f4_0
    afdbench 16885  0.0  0.0  44404  1024 ?        D    13:08   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
    afdbench 16886  0.0  0.0   4064   712 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 2 0 8 e1d94fe0/0/48511fdd_2f4_0
    afdbench 16890  0.0  0.0   4064   716 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd2 2 0 0 bca9d45f/0/48511fdd_2d9_0
    afdbench 16891  0.0  0.0   4064   712 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 0 0 9 26686393/0/48511fdd_2f4_0
    afdbench 16892  0.0  0.0   4064   712 ?        D    13:08   0:00 sf_ftp /home/afdbench/afd1 1 0 10 ae36cee/0/48511fdd_2f4_0

This time there is no OOPS and system is still up running without any
problem (except any process wanting to write something to this filesystem
gets stuck forever).

What can I do to help find the problem? The system is still up with all those
process hanging in D-state.

Thanks,
Holger

Re: Performance of ext4

[Aneesh Kumar K.V]

On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> This time there is no OOPS and system is still up running without any
> problem (except any process wanting to write something to this filesystem
> gets stuck forever).
>
> What can I do to help find the problem? The system is still up with all those
> process hanging in D-state.
>

if you can login to the system get the dmesg output after

echo t > /proc/sysrq-trigger


-aneesh

Re: Performance of ext4

[Holger Kiehl]

On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:

> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>> This time there is no OOPS and system is still up running without any
>> problem (except any process wanting to write something to this filesystem
>> gets stuck forever).
>>
>> What can I do to help find the problem? The system is still up with all those
>> process hanging in D-state.
>>
>
> if you can login to the system get the dmesg output after
>
> echo t > /proc/sysrq-trigger
>
Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.

Holger

Re: Performance of ext4

[Holger Kiehl]

On Thu, 12 Jun 2008, Holger Kiehl wrote:

> On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
>
>> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>>> This time there is no OOPS and system is still up running without any
>>> problem (except any process wanting to write something to this filesystem
>>> gets stuck forever).
>>> 
>>> What can I do to help find the problem? The system is still up with all 
>>> those
>>> process hanging in D-state.
>>> 
>> 
>> if you can login to the system get the dmesg output after
>> 
>> echo t > /proc/sysrq-trigger
>> 
> Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
>
After recompiling, rebooting and run afdbench first I got an OOPS and the
system hanged up solid. The only thing I was able to catch is this:

   RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
   RSP <ffff8101fe259c18>

This was copied by hand. And this I cut 'cut and past' from my terminal:

kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89 df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01

So I rebooted run bonnie and then afdbench and I get the same problem as
yesterday. All process trying to write something to this filesystem hang
in D-state. But now I was able to do the echo t > /proc/sysrq-trigger.
Here the dmesg output:

node_dirty+0x134/0x147
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a715>] hrtick_set+0xa1/0x10a
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

trans_db_log  S 00000000ffffffff     0  8010   8005
  ffff81016f58d9e8 0000000000000082 0000000000000002 0000000000000000
  ffffffff80600cc0 ffff810180001600 0000000000000000 ffff81017f6c1640
  ffff81007fb43210 ffff81017f6c1878 0000000280266f65 ffff81017f6c1878
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
  [<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
  [<ffffffff802fe4d7>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff8025ba69>] find_lock_page+0x29/0x87
  [<ffffffff8025c094>] filemap_fault+0x1cb/0x329
  [<ffffffff80268f59>] __do_fault+0x321/0x369
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8026a7dc>] handle_mm_fault+0x348/0x6a3
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8028e58a>] do_filp_open+0x3f2/0x832
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch S 00000000ffffffff     0  8011   8005
  ffff81017f79b9e8 0000000000000086 ffff810000002618 0000000100000000
  ffffffff80600cc0 0000000000000000 ffff810000002608 ffff81017f6c10b0
  ffffffff8054d350 ffff81017f6c12e8 0000000000001580 ffff81017f6c12e8
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802a3be1>] __find_get_block+0x14c/0x15c
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff802de7f2>] ext4_getblk+0xb4/0x170
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff80294aed>] __d_lookup+0xbd/0x107
  [<ffffffff802e6fbf>] ext4fs_dirhash+0x10d/0x1d2
  [<ffffffff802d9d81>] ext4_htree_store_dirent+0xe6/0xf1
  [<ffffffff802e3745>] htree_dirblock_to_tree+0x12c/0x13e
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8028f91c>] filldir+0x68/0xb7
  [<ffffffff8028f8b4>] filldir+0x0/0xb7
  [<ffffffff802d9938>] ext4_readdir+0x1c1/0x524
  [<ffffffff8028f8b4>] filldir+0x0/0xb7
  [<ffffffff8028e572>] do_filp_open+0x3da/0x832
  [<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
  [<ffffffff802985f0>] mnt_want_write+0x25/0x77
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log     D 00000000ffffffff     0  8012   8005
  ffff81017f79db58 0000000000000086 0000000000000008 0000000000000000
  ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c5370
  ffff81007fb442c0 ffff81017f6c55a8 0000000380290dd5 ffff81017f6c55a8
Call Trace:
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a715>] hrtick_set+0xa1/0x10a
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log    D 00000000ffffffff     0  8013   8005
  ffff81016f7b9b58 0000000000000086 0000000000000008 0000000000000000
  ffffffff80600cc0 0000000000000008 0000000000000000 ffff81017f6c4de0
  ffff81007fb42160 ffff81017f6c5018 0000000180290dd5 ffff81017f6c5018
Call Trace:
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a715>] hrtick_set+0xa1/0x10a
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

delete_log    S 0000000000000000     0  8014   8005
  ffff81017f7099e8 0000000000000086 0000000000000002 0000000000000000
  ffffffff80600cc0 ffff810000002600 0000000000000000 ffff81017f6c26f0
  ffff81017f784850 ffff81017f6c2928 0000000080266f65 ffff81017f6c2928
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
  [<ffffffff80266f65>] zone_statistics+0x3c/0x90
  [<ffffffff8026112c>] get_page_from_freelist+0x461/0x5d5
  [<ffffffff80266f65>] zone_statistics+0x3c/0x90
  [<ffffffff8026112c>] get_page_from_freelist+0x461/0x5d5
  [<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8026a75d>] handle_mm_fault+0x2c9/0x6a3
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8026f46b>] mmap_region+0x41a/0x505
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

production_lo S 00000000ffffffff     0  8015   8005
  ffff81017f70b9e8 0000000000000082 0000000000000002 0000000000000000
  ffffffff80600cc0 ffff810200001600 0000000000000000 ffff81017f6c42c0
  ffff81007fb43210 ffff81017f6c44f8 00000002794b5d20 ffff81017f6c44f8
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802e6dc1>] __ext4_journal_stop+0x1f/0x3d
  [<ffffffff802a0156>] __mark_inode_dirty+0xed/0x187
  [<ffffffff8025c732>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802a0156>] __mark_inode_dirty+0xed/0x187
  [<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
  [<ffffffff80268f59>] __do_fault+0x321/0x369
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8026f46b>] mmap_region+0x41a/0x505
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ S 00000000ffffffff     0  8016   8005
  ffff81016f5959e8 0000000000000086 0000000000080780 ffffe20008ba3f98
  ffffffff80600cc0 ffffffff802a3869 0000fee20000011d ffff81017f6c3d30
  ffff81007fb442c0 ffff81017f6c3f68 00000003794b5d20 ffff81017f6c3f68
Call Trace:
  [<ffffffff802a3869>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802e6dc1>] __ext4_journal_stop+0x1f/0x3d
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff8025c732>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff8023136c>] current_fs_time+0x1e/0x24
  [<ffffffff8025b358>] remove_suid+0x18/0x4b
  [<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a715>] hrtick_set+0xa1/0x10a
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

amg           S 00000000ffffffff     0  8017   8005
  ffff81016f5e59e8 0000000000000082 ffff81023ec9a070 0000000000000282
  ffffffff80600cc0 ffff81007f2e4ac0 0000000000011210 ffff81017f6c37a0
  ffffffff8054d350 ffff81017f6c39d8 0000000000000040 ffff81017f6c39d8
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff80266f65>] zone_statistics+0x3c/0x90
  [<ffffffff80294aed>] __d_lookup+0xbd/0x107
  [<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff80294aed>] __d_lookup+0xbd/0x107
  [<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8028cebc>] __link_path_walk+0xcb1/0xdd9
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8028c1a9>] getname+0x13e/0x1a0
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff8023e876>] remove_wait_queue+0x12/0x41
  [<ffffffff80230135>] do_wait+0xae4/0xb88
  [<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

afdd          S 00000000ffffffff     0  8018   8005
  ffff81016f5e79e8 0000000000000086 ffff810200001618 0000000200000000
  ffffffff80600cc0 0000000000000000 ffff810200001608 ffff81017f6c0000
  ffff81007fb442c0 ffff81017f6c0238 0000000300000b00 ffff81017f6c0238
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff8036750a>] extract_buf+0x84/0xfc
  [<ffffffff80367166>] mix_pool_bytes_extract+0x58/0x148
  [<ffffffff8036750a>] extract_buf+0x84/0xfc
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff80367476>] account+0xd6/0xe6
  [<ffffffff80224a67>] __wake_up+0x38/0x4f
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     D ffff810027160690     0  8019   8017
  ffff81007e8b1d70 0000000000000086 ffff81007e8b1d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8100707910b0
  ffff8100469a2160 ffff8100707912e8 000000028028c8f9 ffff8100707912e8
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8028cebc>] __link_path_walk+0xcb1/0xdd9
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028ac27>] lock_rename+0x35/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat      S 00000000ffffffff     0  8020   8005
  ffff81016f7fd9e8 0000000000000086 0000000000000000 000000000009829c
  ffffffff80600cc0 ffffffff80323d50 ffff810201e124a0 ffff81017f6c0590
  ffff81007fb442c0 ffff81017f6c07c8 0000000300000000 ffff81017f6c07c8
Call Trace:
  [<ffffffff80323d50>] rb_insert_color+0x66/0xe2
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff802fe4d7>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff8023e724>] wake_bit_function+0x0/0x23
  [<ffffffff802ec6aa>] __ext4_journal_dirty_metadata+0x1e/0x46
  [<ffffffff802dbbee>] ext4_mark_iloc_dirty+0x415/0x47e
  [<ffffffff802dca1c>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802db3e9>] walk_page_buffers+0x65/0x8b
  [<ffffffff802db568>] ext4_bh_unmapped+0x0/0xe
  [<ffffffff80324411>] __up_read+0x13/0x8a
  [<ffffffff80322db6>] __prop_inc_single+0x1d/0x36
  [<ffffffff8026941e>] do_wp_page+0x47d/0x4ba
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8022a715>] hrtick_set+0xa1/0x10a
  [<ffffffff80324411>] __up_read+0x13/0x8a
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
  [<ffffffff80286f15>] cp_new_stat+0xe9/0xfc
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

fd            S 00000000ffffffff     0  8021   8005
  ffff81016f5139e8 0000000000000082 0000000000000000 0000000000000000
  ffffffff80600cc0 0000000000000001 0000000180001608 ffff81017f6c3210
  ffff81007fb442c0 ffff81017f6c3448 0000000300000b00 ffff81017f6c3448
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff80294aed>] __d_lookup+0xbd/0x107
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff8028cfc6>] __link_path_walk+0xdbb/0xdd9
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff8028c1a9>] getname+0x13e/0x1a0
  [<ffffffff8028de4c>] __user_walk_fd+0x41/0x4c
  [<ffffffff80286ded>] vfs_stat_fd+0x1b/0x4a
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

show_bench_st S 00000000ffffffff     0  8043   2879
  ffff810023a77ea8 0000000000000082 00007fff24bbfe80 00007f631cabd000
  ffffffff80600cc0 ffffffff80242f85 ffff810023a77ed8 ffff810070793d30
  ffff81007fb43210 ffff810070793f68 0000000200000000 ffff810070793f68
Call Trace:
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff80241154>] hrtimer_start+0x100/0x122
  [<ffffffff80241035>] hrtimer_try_to_cancel+0x61/0x6a
  [<ffffffff80466799>] do_nanosleep+0x64/0xa6
  [<ffffffff802411ca>] hrtimer_nanosleep+0x54/0xbf
  [<ffffffff80240ab2>] hrtimer_wakeup+0x0/0x22
  [<ffffffff80241281>] sys_nanosleep+0x4c/0x62
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

tee           D 0000000000000000     0  8044   2879
  ffff810046939b58 0000000000000086 ffff81001af65240 ffff810049075630
  ffffffff80600cc0 ffffffff80324411 ffff810049075630 ffff8100707942c0
  ffff81007e84de90 ffff8100707944f8 00000000ff8254a0 ffff8100707944f8
Call Trace:
  [<ffffffff80324411>] __up_read+0x13/0x8a
  [<ffffffff80268f59>] __do_fault+0x321/0x369
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8036a969>] tty_write+0x200/0x21b
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? ffff810070796518     0  8413   8019
  ffff81003453bef8 0000000000000046 ffff810070462908 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070796420
  ffff81017f6c42c0 ffff810070796658 0000000200000000 ffff810070796658
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? ffff810070796aa8     0  8415   8019
  ffff8100492c9ef8 0000000000000046 ffff810070462908 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff8100707969b0
  ffff8101ff6fbd30 ffff810070796be8 0000000000000000 ffff810070796be8
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? ffff81016f416aa8     0  8460   7996
  ffff81016f6c1ef8 0000000000000046 ffff810070461048 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f4169b0
  ffff81017f7826f0 ffff81016f416be8 0000000100000000 ffff81016f416be8
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? ffff81016f417038     0  8463   7996
  ffff81016f429ef8 0000000000000046 ffff810070461048 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff81016f416f40
  ffff81027759d370 ffff81016f417178 0000000100000003 ffff81016f417178
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? ffff810070790c18     0  8492   7973
  ffff81002382def8 0000000000000046 ffff8101ff70fb88 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070790b20
  ffff81027f148000 ffff810070790d58 0000000100000000 ffff810070790d58
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     ? 00000000ffffffff     0  8515   7973
  ffff81006e4a7ef8 0000000000000046 ffff8101ff70fb88 0000000000000056
  ffffffff80600cc0 ffffffff80237be4 0000000000000011 ffff810070791640
  ffff81007fb43210 ffff810070791878 0000000200000000 ffff810070791878
Call Trace:
  [<ffffffff80237be4>] do_notify_parent+0x18c/0x199
  [<ffffffff8023096d>] do_exit+0x661/0x665
  [<ffffffff802309d7>] do_group_exit+0x66/0x93
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 0000000000000000     0  9928   8021
  ffff81027851fbe8 0000000000000086 ffff81017f72de90 0000000300000000
  ffffffff80600cc0 0000000000000000 0000000000000000 ffff81027f2bcde0
  ffff81017f6c1bd0 ffff81027f2bd018 0000000100000000 ffff81027f2bd018
Call Trace:
  [<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
  [<ffffffff804674e1>] _read_lock_bh+0x9/0x19
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff8028d093>] path_walk+0xaf/0xb9
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff80282ad0>] __dentry_open+0x141/0x226
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff810052a38000     0  9929   2487
  ffff810046a57bc8 0000000000000086 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a37a0
  ffff8101b97f8590 ffff8100469a39d8 0000000100000001 ffff8100469a39d8
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0  9937   9929
  ffff8101f8e99dd0 0000000000000086 ffffffff8040820c ffff8101f8e99d34
  ffffffff80600cc0 0000000ed348224f ffff81007e8a0ec8 ffff8101b97f8590
  ffff81007fb42160 ffff8101b97f87c8 0000000100000000 ffff8101b97f87c8
Call Trace:
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028d093>] path_walk+0xaf/0xb9
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
  [<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
  [<ffffffff802828e0>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 0000000000000000     0  9945   8021
  ffff81027f295be8 0000000000000082 ffff81007e856290 ffffffff8042e6b5
  ffffffff80600cc0 ffff81027f295b90 ffff8100370343f0 ffff81027f2bd900
  ffff81027f2ba160 ffff81027f2bdb38 0000000037008438 ffff81027f2bdb38
Call Trace:
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d509>] unmap_region+0x10f/0x125
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff81016f680000     0  9946   2487
  ffff8100706ddbc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8100469a5900
  ffff81016f4137a0 ffff8100469a5b38 0000000000000001 ffff8100469a5b38
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0  9949   9946
  ffff810147d79dd0 0000000000000082 ffffffff8040820c ffff810147d79d34
  ffffffff80600cc0 0000000ed34681af ffff81007e8a0ec8 ffff81016f4137a0
  ffffffff8054d350 ffff81016f4139d8 0000000000000000 ffff81016f4139d8
Call Trace:
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028d093>] path_walk+0xaf/0xb9
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
  [<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
  [<ffffffff802828e0>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff8101ff703c80     0  9959   2487
  ffff8101865c7bc8 0000000000000086 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97ff4d0
  ffff81016f4137a0 ffff8101b97ff708 0000000000000001 ffff8101b97ff708
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff81007e85a400     0  9961   9959
  ffff8101b9793c08 0000000000000082 000000000002842a ffff8101b9793d84
  ffffffff80600cc0 0000000000000000 ffff81027a138ea0 ffff8101b97f90b0
  ffff8101b97ff4d0 ffff8101b97f92e8 0000000000000001 ffff8101b97f92e8
Call Trace:
  [<ffffffff802e6fbf>] ext4fs_dirhash+0x10d/0x1d2
  [<ffffffff803019dd>] jbd2_journal_release_jbd_inode+0xaf/0x100
  [<ffffffff8023e724>] wake_bit_function+0x0/0x23
  [<ffffffff802d7d9b>] ext4_discard_reservation+0x1e/0x5a
  [<ffffffff80296488>] clear_inode+0x69/0xc0
  [<ffffffff80296afd>] generic_drop_inode+0x13a/0x157
  [<ffffffff802db069>] ext4_new_inode+0xc46/0xcac
  [<ffffffff802feabf>] start_this_handle+0x2c7/0x370
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff802e251d>] ext4_create+0x88/0x105
  [<ffffffff802e27e7>] ext4_lookup+0x97/0xc1
  [<ffffffff8028b7d9>] vfs_create+0x7d/0xed
  [<ffffffff8028e385>] do_filp_open+0x1ed/0x832
  [<ffffffff802828e0>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 0000000000000000     0  9969   8021
  ffff8102785bfbe8 0000000000000082 ffff81017f72dcd0 ffffffff8042e6b5
  ffffffff80600cc0 0000000000000000 0000000000000000 ffff81027f2bbd30
  ffff81017f6c1bd0 ffff81027f2bbf68 000000019740516a ffff81027f2bbf68
Call Trace:
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d509>] unmap_region+0x10f/0x125
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S 00000000ffffffff     0  9970   2487
  ffff810224881bc8 0000000000000082 00000000008db1fc ffffffff80241310
  ffffffff80600cc0 0000000000000001 ffffffff80266f65 ffff81027f2bb7a0
  ffff81007fb442c0 ffff81027f2bb9d8 000000037f2bc888 ffff81027f2bb9d8
Call Trace:
  [<ffffffff80241310>] ktime_get_ts+0x21/0x4a
  [<ffffffff80266f65>] zone_statistics+0x3c/0x90
  [<ffffffff80266f65>] zone_statistics+0x3c/0x90
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8021eee3>] do_page_fault+0x60c/0x825
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff80467699>] error_exit+0x0/0x51
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff81027a138f60     0  9973   9970
  ffff81022cadfdd0 0000000000000086 0000000000000000 ffff810200000000
  ffffffff80600cc0 0000000f13fc4c86 ffff81007e8a0ec8 ffff81027f2bf4d0
  ffff81027f2bb7a0 ffff81027f2bf708 0000000300000000 ffff81027f2bf708
Call Trace:
  [<ffffffff8028d093>] path_walk+0xaf/0xb9
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028e2a4>] do_filp_open+0x10c/0x832
  [<ffffffff8028280b>] get_unused_fd_flags+0x7f/0x10e
  [<ffffffff802828e0>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff810027160690     0  9979   8021
  ffff81022e071d70 0000000000000086 ffff81022e071d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff81027f2bc850
  ffff81016f416420 ffff81027f2bca88 000000018028c8f9 ffff81027f2bca88
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028ac27>] lock_rename+0x35/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff81016f683c80     0  9981   2487
  ffff81016f4f9bc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f416420
  ffff81017f6c4de0 ffff81016f416658 0000000100000001 ffff81016f416658
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S 00007f17790fc000     0  9984   9981
  ffff81016f555b68 0000000000000086 ffff81017f72de90 ffffffff802414e9
  ffffffff80600cc0 000000000f4c63de 0000000000000046 ffff81016f417a60
  ffff81027f2bc850 ffff81016f417c98 000000017f72de90 ffff81016f417c98
Call Trace:
  [<ffffffff802414e9>] ktime_get_real+0xc/0x43
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff80409265>] sk_wait_data+0x83/0xc6
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
  [<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
  [<ffffffff80241154>] hrtimer_start+0x100/0x122
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff810027160690     0 10194   7998
  ffff810186669d70 0000000000000086 ffff810186669d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fb210
  ffff81027f149bd0 ffff8101b97fb448 000000028028c8f9 ffff8101b97fb448
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028ac27>] lock_rename+0x35/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff8101fff33700     0 10195   2487
  ffff81016f625bc8 0000000000000086 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f414850
  ffff8101b97fef40 ffff81016f414a88 0000000200000001 ffff81016f414a88
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S 00007f17790fc000     0 10197  10195
  ffff8101ff7f1b68 0000000000000086 ffff8101867b4450 ffffffff802414e9
  ffffffff80600cc0 000000000f5245fe 0000000000000046 ffff8101b97fd900
  ffff8101b97fb210 ffff8101b97fdb38 00000002867b4450 ffff8101b97fdb38
Call Trace:
  [<ffffffff802414e9>] ktime_get_real+0xc/0x43
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff80409265>] sk_wait_data+0x83/0xc6
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
  [<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
  [<ffffffff80241154>] hrtimer_start+0x100/0x122
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        S 0000000000000000     0 10198   7975
  ffff81007054f9e8 0000000000000086 ffff8101f8f1e310 9c9ba13000000834
  ffffffff80600cc0 000000038023e617 ffff8101db55d930 ffff8100469a5370
  ffff8100469a10b0 ffff8100469a55a8 00000002867b5cd0 ffff8100469a55a8
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
  [<ffffffff804674e1>] _read_lock_bh+0x9/0x19
  [<ffffffffa00634f5>] :ip_tables:ipt_do_table+0x2e9/0x30c
  [<ffffffffa006bd34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
  [<ffffffffa007937e>] :nf_conntrack_ipv4:ipv4_confirm+0xcc/0xd7
  [<ffffffff80425272>] nf_iterate+0x41/0x7d
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff8042ea4b>] ip_finish_output+0x0/0x239
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff802985f0>] mnt_want_write+0x25/0x77
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80408dd6>] lock_sock_nested+0x9a/0xa5
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
  [<ffffffff8023e6e5>] wake_up_bit+0x11/0x22
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff810052a3b700     0 10199   2487
  ffff810147c4fbc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 ffff810181e11cc0 ffff81016f412160
  ffff8100469a10b0 ffff81016f412398 0000000200000001 ffff81016f412398
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0 10201  10199
  ffff810046851b58 0000000000000082 ffff8101fff7b200 0000000000000292
  ffffffff80600cc0 ffffffff80411a78 ffff81018670386c ffff8100469a10b0
  ffff81007fb43210 ffff8100469a12e8 0000000200000246 ffff8100469a12e8
Call Trace:
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80409110>] sk_reset_timer+0xf/0x19
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8025cfb5>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff810027160690     0 10202   7998
  ffff8101b9727d70 0000000000000082 ffff8101b9727d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fc2c0
  ffff81016f4126f0 ffff8101b97fc4f8 000000038028c8f9 ffff8101b97fc4f8
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028ac27>] lock_rename+0x35/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff810052a3a3c0     0 10203   2487
  ffff810147c47bc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff81016f4126f0
  ffff81027f2b8b20 ffff81016f412928 0000000300000001 ffff81016f412928
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 10205   7998
  ffff8101f8f71d70 0000000000000082 ffff8101f8f71d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97f8000
  ffffffff8054d350 ffff8101b97f8238 000000008028c8f9 ffff8101b97f8238
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028ac27>] lock_rename+0x35/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff81016f681b80     0 10206   2487
  ffff81017f769bc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 ffff810001006cc0 ffff81016f410b20
  ffff8101b97fd370 ffff81016f410d58 0000000300000001 ffff81016f410d58
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S 00007f17790fc000     0 10208  10203
  ffff810034705b68 0000000000000086 ffff810246fa5410 ffffffff802414e9
  ffffffff80600cc0 000000000f5342a7 0000000000000046 ffff8100469a5e90
  ffff8101b97fc2c0 ffff8100469a60c8 0000000346fa5410 ffff8100469a60c8
Call Trace:
  [<ffffffff802414e9>] ktime_get_real+0xc/0x43
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff80409265>] sk_wait_data+0x83/0xc6
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
  [<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
  [<ffffffff80241154>] hrtimer_start+0x100/0x122
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S 00007f17790fc000     0 10209  10206
  ffff810147d37b68 0000000000000086 ffff81007e857b10 ffffffff802414e9
  ffffffff80600cc0 000000000f1a1bb0 0000000000000046 ffff81016f413210
  ffff8101b97f8000 ffff81016f413448 000000007e857b10 ffff81016f413448
Call Trace:
  [<ffffffff802414e9>] ktime_get_real+0xc/0x43
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff80409265>] sk_wait_data+0x83/0xc6
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80434547>] tcp_recvmsg+0x3b3/0x7d2
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80406a49>] sock_recvmsg+0xd5/0xed
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff80406509>] sockfd_lookup_light+0x1a/0x52
  [<ffffffff80407a7d>] sys_recvfrom+0xc3/0x122
  [<ffffffff80241154>] hrtimer_start+0x100/0x122
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 10210   7975
  ffff81001ac6fbe8 0000000000000086 ffff8101867b4d10 000000038042e6b5
  ffffffff80600cc0 0000000000000246 ffff81007fbbb000 ffff8100469a2160
  ffff81007fb43210 ffff8100469a2398 0000000200000000 ffff8100469a2398
Call Trace:
  [<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
  [<ffffffff804674e1>] _read_lock_bh+0x9/0x19
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff8028d093>] path_walk+0xaf/0xb9
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff80282ad0>] __dentry_open+0x141/0x226
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff8101fff32100     0 10211   2487
  ffff8101b960bbc8 0000000000000086 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fa160
  ffff8101b97fb210 ffff8101b97fa398 0000000200000001 ffff8101b97fa398
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff8101dbe48f60     0 10213  10211
  ffff8101f8ff3d70 0000000000000082 ffff8101f8ff3d38 ffff81007e8a0e00
  ffffffff80600cc0 0000000000000000 ffffffff80293dad ffff8101b97fef40
  ffff8100469a10b0 ffff8101b97ff178 000000028028c8f9 ffff8101b97ff178
Call Trace:
  [<ffffffff80293dad>] dput+0x21/0x10a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff804665e8>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466484>] mutex_lock+0xa/0xb
  [<ffffffff8028acaf>] lock_rename+0xbd/0xc5
  [<ffffffff8028d51e>] do_rename+0x98/0x1b2
  [<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 0000000000000000     0 10214   7975
  ffff810046addbe8 0000000000000082 ffff81017f72cb50 ffffffff8042e6b5
  ffffffff80600cc0 0000000000000246 ffff81007fbbb000 ffff8100469a3d30
  ffff81027f14de90 ffff8100469a3f68 000000017f6a2a6c ffff8100469a3f68
Call Trace:
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022595b>] check_preempt_wakeup+0x7f/0xbc
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d509>] unmap_region+0x10f/0x125
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff810220725e40     0 10215   2487
  ffff8101866ffbc8 0000000000000086 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fe420
  ffff81027f2b90b0 ffff8101b97fe658 0000000100000001 ffff8101b97fe658
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 0000000000000000     0 10217  10215
  ffff810220501bf8 0000000000000086 0000000000000001 0000000000000000
  ffffffff80600cc0 0000000000000000 ffff810220501ba8 ffff81027f2b90b0
  ffff81017f781bd0 ffff81027f2b92e8 0000000100000000 ffff81027f2b92e8
Call Trace:
  [<ffffffff80224a67>] __wake_up+0x38/0x4f
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e2b2c>] ext4_rename+0x4d/0x64a
  [<ffffffff8040820c>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028bc30>] vfs_rename+0x1fd/0x376
  [<ffffffff8028d5d7>] do_rename+0x151/0x1b2
  [<ffffffff80261447>] __alloc_pages_internal+0xd2/0x40c
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d684>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 10218   7998
  ffff8101866c1be8 0000000000000082 ffff810246fa4610 ffffffff8042e6b5
  ffffffff80600cc0 ffff8101866c1b90 ffff8101db68f7e0 ffff8101b97fc850
  ffff81007fb442c0 ffff8101b97fca88 000000034f9ed590 ffff8101b97fca88
Call Trace:
  [<ffffffff8042e6b5>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff802a3c0f>] __getblk+0x1e/0x28d
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8025ba69>] find_lock_page+0x29/0x87
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff8028a261>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802736e8>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d509>] unmap_region+0x10f/0x125
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff81016f682100     0 10219   2487
  ffff8101f8ea7bc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 ffff810201e11cc0 ffff8101b97fde90
  ffff81027f2b90b0 ffff8101b97fe0c8 0000000100000001 ffff8101b97fe0c8
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8045567a>] fn_hash_lookup+0x83/0xc1
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        S 0000000000000000     0 10220   7975
  ffff81016f4859e8 0000000000000082 ffff81023b55c8b0 9c34adbf00000834
  ffffffff80600cc0 000000038023e617 ffff8101db55d930 ffff81016f415900
  ffff81027f2b8b20 ffff81016f415b38 0000000346fa5250 ffff81016f415b38
Call Trace:
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffff80235889>] __mod_timer+0xaa/0xb8
  [<ffffffff80466249>] schedule_timeout+0xa6/0xc9
  [<ffffffff80235392>] process_timeout+0x0/0x5
  [<ffffffff80290728>] do_select+0x414/0x482
  [<ffffffff80290dd5>] __pollwait+0x0/0xe3
  [<ffffffff80224eed>] default_wake_function+0x0/0xe
  [<ffffffff80411a78>] dev_queue_xmit+0x21a/0x232
  [<ffffffff802356ef>] lock_timer_base+0x26/0x4b
  [<ffffffffa0068801>] :nf_conntrack:nf_conntrack_in+0x455/0x508
  [<ffffffff804674e1>] _read_lock_bh+0x9/0x19
  [<ffffffffa00634f5>] :ip_tables:ipt_do_table+0x2e9/0x30c
  [<ffffffffa006bd34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
  [<ffffffffa007937e>] :nf_conntrack_ipv4:ipv4_confirm+0xcc/0xd7
  [<ffffffff80425272>] nf_iterate+0x41/0x7d
  [<ffffffff80242f85>] getnstimeofday+0x38/0x84
  [<ffffffff8042ea4b>] ip_finish_output+0x0/0x239
  [<ffffffff80290941>] core_sys_select+0x1ab/0x253
  [<ffffffff80231c26>] local_bh_enable+0x6a/0x7c
  [<ffffffff802985f0>] mnt_want_write+0x25/0x77
  [<ffffffff80295a4e>] touch_atime+0x12/0xfa
  [<ffffffff80408dd6>] lock_sock_nested+0x9a/0xa5
  [<ffffffff804674fa>] _spin_lock_bh+0x9/0x1f
  [<ffffffff80408cba>] release_sock+0x13/0x95
  [<ffffffff8023e617>] bit_waitqueue+0x10/0xa0
  [<ffffffff8023e6e5>] wake_up_bit+0x11/0x22
  [<ffffffff80290d25>] sys_select+0xb7/0x167
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        S ffff810220726680     0 10221   2487
  ffff810195039bc8 0000000000000082 0000000000000096 ffffffff80224edc
  ffffffff80600cc0 0000000000000001 0000000000000096 ffff8101b97fd370
  ffff81016f410590 ffff8101b97fd5a8 0000000300000001 ffff8101b97fd5a8
Call Trace:
  [<ffffffff80224edc>] try_to_wake_up+0x155/0x166
  [<ffffffff8040da2f>] memcpy_fromiovec+0x34/0x63
  [<ffffffff8040bc6a>] skb_queue_tail+0x17/0x3e
  [<ffffffff80409098>] sock_def_readable+0x10/0x5d
  [<ffffffff80459e5c>] unix_stream_sendmsg+0x293/0x353
  [<ffffffff804661c1>] schedule_timeout+0x1e/0xc9
  [<ffffffff8023e81a>] prepare_to_wait+0x15/0x5f
  [<ffffffff804598c9>] unix_stream_recvmsg+0x282/0x582
  [<ffffffff80224b2b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80405f35>] sock_aio_read+0x107/0x112
  [<ffffffff80406daf>] sys_sendmsg+0x217/0x28a
  [<ffffffff80283f9a>] do_sync_read+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802848d7>] vfs_read+0xbd/0x132
  [<ffffffff80284a08>] sys_read+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 0000000000000000     0 10224  10221
  ffff810246ef1a68 0000000000000086 000000003b55c7e0 ffffffff80233c4d
  ffffffff80600cc0 ffffffff802d7ba2 0000000000000003 ffff81027f2b8b20
  ffff8101b97fd370 ffff81027f2b8d58 0000000334b600c0 ffff81027f2b8d58
Call Trace:
  [<ffffffff80233c4d>] __capable+0x9/0x1f
  [<ffffffff802d7ba2>] ext4_has_free_blocks+0x2b/0xad
  [<ffffffff802a3869>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802a4264>] generic_write_end+0x5d/0x68
  [<ffffffff8025c6bc>] generic_file_buffered_write+0x1b6/0x658
  [<ffffffff80409110>] sk_reset_timer+0xf/0x19
  [<ffffffff8043cd07>] tcp_rcv_established+0x908/0xbc5
  [<ffffffff8023136c>] current_fs_time+0x1e/0x24
  [<ffffffff8025b358>] remove_suid+0x18/0x4b
  [<ffffffff8025d09a>] __generic_file_aio_write_nolock+0x34a/0x37e
  [<ffffffff8025d799>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9ed8>] ext4_file_write+0x96/0x11a
  [<ffffffff80283e87>] do_sync_write+0xce/0x113
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846ac>] vfs_write+0xad/0x136
  [<ffffffff802847f1>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0 10225  10219
  ffff810147d55ce8 0000000000000086 ffff8101a2dcadd8 0000000000000006
  ffffffff80600cc0 ffff8102796cabd0 ffff810274479000 ffff81016f410590
  ffff81007fb442c0 ffff81016f4107c8 000000030000000f ffff81016f4107c8
Call Trace:
  [<ffffffff8028ad6a>] do_lookup+0x63/0x1a1
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e24e1>] ext4_create+0x4c/0x105
  [<ffffffff802e27e7>] ext4_lookup+0x97/0xc1
  [<ffffffff8028b7d9>] vfs_create+0x7d/0xed
  [<ffffffff8028e385>] do_filp_open+0x1ed/0x832
  [<ffffffff802828e0>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 10226   8021
  ffff81027f0afbb8 0000000000000082 0000003df881a530 ffff81003726f1c0
  ffffffff80600cc0 ffffffff80260923 0000000000000002 ffff81027f2bfa60
  ffff81007fb42160 ffff81027f2bfc98 000000010000001e ffff81027f2bfc98
Call Trace:
  [<ffffffff80260923>] __rmqueue_smallest+0x9d/0x124
  [<ffffffff802609c7>] __rmqueue+0x1d/0x1fb
  [<ffffffff802feaa5>] start_this_handle+0x2ad/0x370
  [<ffffffff8023e6f6>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecca>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dcb2b>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a0092>] __mark_inode_dirty+0x29/0x187
  [<ffffffff802959f7>] file_update_time+0xaf/0xf4
  [<ffffffff802693fc>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff8028f8b4>] filldir+0x0/0xb7
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff8029260e>] fcntl_setlk+0x285/0x296
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467699>] error_exit+0x0/0x51

Re: Performance of ext4

[Jan Kara]

> On Thu, 12 Jun 2008, Holger Kiehl wrote:
> 
> >On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
> >
> >>On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> >>>This time there is no OOPS and system is still up running without any
> >>>problem (except any process wanting to write something to this filesystem
> >>>gets stuck forever).
> >>>
> >>>What can I do to help find the problem? The system is still up with all 
> >>>those
> >>>process hanging in D-state.
> >>>
> >>
> >>if you can login to the system get the dmesg output after
> >>
> >>echo t > /proc/sysrq-trigger
> >>
> >Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> >reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
> >
> After recompiling, rebooting and run afdbench first I got an OOPS and the
> system hanged up solid. The only thing I was able to catch is this:
> 
>   RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
>   RSP <ffff8101fe259c18>
> 
> This was copied by hand. And this I cut 'cut and past' from my terminal:
> 
> kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89 
> df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48> 
> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
  Aneesh found cause of this oops I think... Aneesh, would you send the
fix to Holger? Thanks.

> So I rebooted run bonnie and then afdbench and I get the same problem as
> yesterday. All process trying to write something to this filesystem hang
> in D-state. But now I was able to do the echo t > /proc/sysrq-trigger.
> Here the dmesg output:
  Sadly, the output seems to be truncated (mainly, the kjournald process
is missing, which is probably the root cause of the hang). So could you
have a look whether /var/log/messages doesn't contain the dump of all
processes? And if no, then could you do "echo w >/proc/sysrq-trigger"
please? That will dump only blocked processes which should fit in the log
buffer. You can also increase log buffer size in kernel config but that
shouldn't be needed. Thanks.

									Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: Performance of ext4

[Aneesh Kumar K.V]

On Mon, Jun 16, 2008 at 07:54:08PM +0200, Jan Kara wrote:
> > On Thu, 12 Jun 2008, Holger Kiehl wrote:
> > 
> > >On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
> > >
> > >>On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
> > >>>This time there is no OOPS and system is still up running without any
> > >>>problem (except any process wanting to write something to this filesystem
> > >>>gets stuck forever).
> > >>>
> > >>>What can I do to help find the problem? The system is still up with all 
> > >>>those
> > >>>process hanging in D-state.
> > >>>
> > >>
> > >>if you can login to the system get the dmesg output after
> > >>
> > >>echo t > /proc/sysrq-trigger
> > >>
> > >Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
> > >reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
> > >
> > After recompiling, rebooting and run afdbench first I got an OOPS and the
> > system hanged up solid. The only thing I was able to catch is this:
> > 
> >   RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
> >   RSP <ffff8101fe259c18>
> > 
> > This was copied by hand. And this I cut 'cut and past' from my terminal:
> > 
> > kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89 
> > df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48> 
> > 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
>   Aneesh found cause of this oops I think... Aneesh, would you send the
> fix to Holger? Thanks.
> 

That fix was mainly done with the help of Holger. Many thanks to him for
doing multiple test during weekend with different combination. He had
already confirmed that the fix worked for me. 

There is another issue that he is hitting when running the test with
mke2fs -m 0. But I think that is not related to lock inversion.

-aneesh

Re: Performance of ext4

[Holger Kiehl]

On Mon, 16 Jun 2008, Aneesh Kumar K.V wrote:

> On Mon, Jun 16, 2008 at 07:54:08PM +0200, Jan Kara wrote:
>>> On Thu, 12 Jun 2008, Holger Kiehl wrote:
>>>
>>>> On Thu, 12 Jun 2008, Aneesh Kumar K.V wrote:
>>>>
>>>>> On Thu, Jun 12, 2008 at 02:07:30PM +0000, Holger Kiehl wrote:
>>>>>> This time there is no OOPS and system is still up running without any
>>>>>> problem (except any process wanting to write something to this filesystem
>>>>>> gets stuck forever).
>>>>>>
>>>>>> What can I do to help find the problem? The system is still up with all
>>>>>> those
>>>>>> process hanging in D-state.
>>>>>>
>>>>>
>>>>> if you can login to the system get the dmesg output after
>>>>>
>>>>> echo t > /proc/sysrq-trigger
>>>>>
>>>> Unfortunately I have not set CONFIG_MAGIC_SYSRQ. Tomorrow I will try to
>>>> reproduce this with a kernel that has CONFIG_MAGIC_SYSRQ set.
>>>>
>>> After recompiling, rebooting and run afdbench first I got an OOPS and the
>>> system hanged up solid. The only thing I was able to catch is this:
>>>
>>>   RIP [<ffffffff803019f9>] jbd2_journal_release_jbd_inode+0xcb/0x100
>>>   RSP <ffff8101fe259c18>
>>>
>>> This was copied by hand. And this I cut 'cut and past' from my terminal:
>>>
>>> kernel: Code: c3 e8 31 ce f3 ff 41 fe 04 24 e8 fe 3f 16 00 4c 89 fe 48 89
>>> df e8 5f cd f3 ff eb 82 48 83 7d 00 00 74 27 48 8b 55 10 48 8b 45 18 <48>
>>> 89 42 08 48 89 10 48 c7 45 18 00 02 20 00 48 c7 45 10 00 01
>>   Aneesh found cause of this oops I think... Aneesh, would you send the
>> fix to Holger? Thanks.
>>
>
> That fix was mainly done with the help of Holger. Many thanks to him for
> doing multiple test during weekend with different combination. He had
> already confirmed that the fix worked for me.
>
> There is another issue that he is hitting when running the test with
> mke2fs -m 0. But I think that is not related to lock inversion.
>
Doing several test with '-m 0' I was unable to reproduce this and I could
now do several runs with afdbench. However the results do show that with
ext4-patch-queue it actually slower:

For ext3:             5449.76 files per second 15.58 MiB/s
For ext4:             5162.16 files per second 15.48 MiB/s
For ext4+patch-queue: 4963.6975 files per second 14.73 MiB/s

On the positive side the bonnie++ numbers got much better:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ext3            16G 51501  97 210601 91 100479 32 55528  98 301589 44  1198   5
                 16G 52702  98 215540 94 99339  32 55376  97 300933 44  1159   4
                 16G 52426  99 212584 94 99091  31 55656  98 301669 44  1160   4
ext4            16G 52965  98 224199 89 108440 32 56389  99 303792 42  1526   4
                 16G 52792  98 223980 92 107685 32 56350  98 303066 42  1532   4
                 16G 52994  98 222354 92 107802 32 56386  99 303727 41  1455   4
ext4(patchqueue)16G 59727  98 252733 52 110177 25 55821  98 296739 42  1553   5
                 16G 61047  99 239242 48 111664 25 55706  98 297151 42  1545   4
                 16G 60503  99 241532 47 109655 25 55671  98 297648 42  1552   3

ext3 and ext4 tests where done with 2.6.25.4 and those with patch-queue was
2.6.26-rc5. I will do another test run with 2.6.26-rc5 without patch-queue
just to make sure that the slowdown does not happen due to changes in the
2.6.26 branch.

Another important observation is that ext4+patch-queue does still have the
same error I reported 2 years ago, see:
     http://marc.info/?l=linux-ext4&m=115192918624449&w=2

During afdbench the results are written to $HOME/results, see the following
listing before running afdbench:

    afdbench@helena:~/results$ ls -ltr
    total 260
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 09:40 results.10740.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1320 2008-05-26 09:47 results.15132.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1593 2008-05-26 10:03 results.20899.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1499 2008-05-26 10:20 results.30577.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1319 2008-05-26 10:33 results.9819.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 10:40 results.16657.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 10:48 results.21319.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 10:59 results.25168.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 11:05 results.29113.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 11:15 results.1626.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 11:47 results.6247.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 11:53 results.10889.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 12:09 results.8970.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench   838 2008-05-26 12:22 results.2635.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1281 2008-05-26 12:35 results.2686.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1327 2008-05-26 12:43 results.7525.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1693 2008-05-26 12:56 results.13665.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1600 2008-05-26 13:15 results.23610.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1600 2008-05-26 15:10 results.13041.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1604 2008-05-26 15:23 results.22862.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1604 2008-05-26 15:43 results.32516.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14582 2008-05-28 17:16 results.2585.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14583 2008-05-28 20:17 results.32490.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14628 2008-05-28 23:19 results.11812.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14845 2008-05-29 02:22 results.31472.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 21134 2008-05-29 05:30 results.27636.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 21970 2008-05-29 08:40 results.25944.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  3309 2008-05-29 11:11 results.14789.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  3310 2008-05-29 11:43 results.20842.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4107 2008-06-16 21:45 results.2190.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4108 2008-06-16 23:04 results.8794.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4108 2008-06-17 00:22 results.27273.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4109 2008-06-17 01:41 results.4505.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  8450 2008-06-17 03:05 results.19995.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  9230 2008-06-17 04:35 results.24033.helena.dwd.de

And here is the listing after running the benchmark:

    afdbench@helena:~/results$ ls -ltr
    total 284
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 09:40 results.10740.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1320 2008-05-26 09:47 results.15132.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1593 2008-05-26 10:03 results.20899.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1499 2008-05-26 10:20 results.30577.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1319 2008-05-26 10:33 results.9819.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 10:40 results.16657.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 10:48 results.21319.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 10:59 results.25168.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 11:05 results.29113.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 11:15 results.1626.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 11:47 results.6247.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1273 2008-05-26 11:53 results.10889.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1274 2008-05-26 12:09 results.8970.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench   838 2008-05-26 12:22 results.2635.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1281 2008-05-26 12:35 results.2686.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1327 2008-05-26 12:43 results.7525.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1693 2008-05-26 12:56 results.13665.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1600 2008-05-26 13:15 results.23610.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1600 2008-05-26 15:10 results.13041.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1604 2008-05-26 15:23 results.22862.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  1604 2008-05-26 15:43 results.32516.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14582 2008-05-28 17:16 results.2585.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14583 2008-05-28 20:17 results.32490.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14628 2008-05-28 23:19 results.11812.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 14845 2008-05-29 02:22 results.31472.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 21134 2008-05-29 05:30 results.27636.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench 21970 2008-05-29 08:40 results.25944.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  3309 2008-05-29 11:11 results.14789.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  3310 2008-05-29 11:43 results.20842.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4107 2008-06-16 21:45 results.2190.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4108 2008-06-16 23:04 results.8794.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4108 2008-06-17 00:22 results.27273.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  4109 2008-06-17 01:41 results.4505.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  8450 2008-06-17 03:05 results.19995.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  8208 2008-06-17 04:35 results.24033.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  8361 2008-06-17 08:19 results.9628.helena.dwd.de
    -rw-r--r-- 1 afdbench afdbench  8858 2008-06-17 09:49 results.8134.helena.dwd.de

Note how the size of file results.24033.helena.dwd.de changes from 9230 before
the test to 8208 bytes after the test. Also note the date both have the same
timestamp "2008-06-17 04:35". I have made a copy of results.24033.helena.dwd.de
before the test and compared it with that after the test. The file is just
truncated by 1022 bytes and there is no garbage.

This is reproducible but not always, sometimes only a small part gets truncated
and sometimes several kilobytes get truncated.

Holger

Re: Performance of ext4

[Holger Kiehl]

On Tue, 17 Jun 2008, Holger Kiehl wrote:

> Doing several test with '-m 0' I was unable to reproduce this and I could
> now do several runs with afdbench. However the results do show that with
> ext4-patch-queue it actually slower:
>
> For ext3:             5449.76 files per second 15.58 MiB/s
> For ext4:             5162.16 files per second 15.48 MiB/s
> For ext4+patch-queue: 4963.6975 files per second 14.73 MiB/s
>
> On the positive side the bonnie++ numbers got much better:
>
> Version  1.03       ------Sequential Output------ --Sequential Input- 
> --Random-
>                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec 
> %CP
> ext3            16G 51501  97 210601 91 100479 32 55528  98 301589 44  1198 
> 5
>                16G 52702  98 215540 94 99339  32 55376  97 300933 44  1159 
> 4
>                16G 52426  99 212584 94 99091  31 55656  98 301669 44  1160 
> 4
> ext4            16G 52965  98 224199 89 108440 32 56389  99 303792 42  1526 
> 4
>                16G 52792  98 223980 92 107685 32 56350  98 303066 42  1532 
> 4
>                16G 52994  98 222354 92 107802 32 56386  99 303727 41  1455 
> 4
> ext4(patchqueue)16G 59727  98 252733 52 110177 25 55821  98 296739 42  1553 
> 5
>                16G 61047  99 239242 48 111664 25 55706  98 297151 42  1545 
> 4
>                16G 60503  99 241532 47 109655 25 55671  98 297648 42  1552 
> 3
>
> ext3 and ext4 tests where done with 2.6.25.4 and those with patch-queue was
> 2.6.26-rc5. I will do another test run with 2.6.26-rc5 without patch-queue
> just to make sure that the slowdown does not happen due to changes in the
> 2.6.26 branch.
>
Here the results without patch queue:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                 16G 52133  98 221378 95 106873 32 55707  99 297065 42  1546   4
                 16G 52042  98 220931 93 107715 32 55939  98 298810 42  1543   3
                 16G 52975  98 220976 93 108060 31 56426  98 298906 42  1452   4

For afdbench: 5336.41 files per second 15.63 MiB/s

So it seems that for afdbench the ext4-patch-queue is a slowdown.

I forgot to mention that for bonnie ext4-patch-queue reduces CPU-load
a lot. For block writting it is nearly halved.

Holger

Re: Performance of ext4

[Andreas Dilger]

On Jun 18, 2008  05:58 +0000, Holger Kiehl wrote:
>      1.03       ------Sequential Output------ --Sequential Input- --Random-
>                 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> ext4               K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>                16G 52133  98 221378 95 106873 32 55707  99 297065 42  1546   4
>                16G 52042  98 220931 93 107715 32 55939  98 298810 42  1543   3
>                16G 52975  98 220976 93 108060 31 56426  98 298906 42  1452   4
>
>ext4(patchqueue)16G 59727  98 252733 52 110177 25 55821  98 296739 42  1553   5
>>               16G 61047  99 239242 48 111664 25 55706  98 297151 42  1545   4
>>               16G 60503  99 241532 47 109655 25 55671  98 297648 42  1552   3
>
> I forgot to mention that for bonnie ext4-patch-queue reduces CPU-load
> a lot. For block writting it is nearly halved.

That was the main reason for developing mballoc.  With this patch it
would be able to drive almost 500MB/s write, 600MB/s read on the above
system instead of being CPU limited at 250/300MB/s.


Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: Performance of ext4

[Theodore Tso]

On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
> For afdbench: 5336.41 files per second 15.63 MiB/s
>
> So it seems that for afdbench the ext4-patch-queue is a slowdown.

Can you remind me where afdbench can be downloaded?  And if I remember
correctly, it creates and deletes large numbers of small files,
correct?

It would be interesting to see which new feature introduced by the
ext4 patch queue --- probably dellayed allocation or mballoc --- is
responsible for the slowdown.  One or the other (or both) can be
disabled by mounting the filesystem (using a kernel with the ext4
patch queue) with the mount options -O nomballoc or -O nodelalloc.

If it turns out that nomballoc restores the speed for afdbench, for
example, then it will tell us where we need to look more closely.
Ideally we would not want to have one mount option needed to optimize
filesystem operations for large amoutns of modifications to small
files, and another mode of operation when mostly writing to large
files.  So if you could do a round of tests using the ext4 patch queue
kernel, with -O nomballoc and -O nodelalloc (and if both seem to
improve things, try "-O nomballoc,nodelalloc" and see if you get back
to the pre-ext4 patch queue speed), it would be very much appreciated.

Thanks, regards,

						- Ted

Re: Performance of ext4

[Holger Kiehl]

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>
>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>
> Can you remind me where afdbench can be downloaded?  And if I remember
> correctly, it creates and deletes large numbers of small files,
> correct?
>
Yes and yes.

You can download the benchmark but it is complicated to setup. You can
download it from ftp://ftp.dwd.de/pub/afd/afdbench-2.0.0.tar.bz2 and
you will also need ftp://ftp.dwd.de/pub/afd/development/afd-1.4.0pre1.tar.bz2

Here a short guide how you need to set this up (there is also a SETUP
file in afdbench-2.0.0.tar.bz2):

    - create a new user for example afdbench
    - untar afdbench-2.0.0.tar.bz2 where ever your test filesystem is mounted
      eg /home
    - ln -s /home/afdbench-2.0.0 /home/afdbench
    - ensure that in .bash_profile of user afdbench you have:
         PATH=$PATH:$HOME/bin
    - login as afdbench
    - Untar afd-1.4.0pre1.tar.bz2
    - cd afd-1.4.0pre1
    - ./configure --prefix=$HOME --enable-ftp_reuse_data_port --enable-passwd_in_msg --enable-expand_path_in_message --enable-compiler-optimizations --enable-with_afdbench_settings --enable-splice_support --enable-sendfile_support
    - make
    - make install-strip
    - In afdbench script change BENCH_PASSWD to whatever you have set the
      password of user afdbench.

If you have problems because you do not have openmotif or lesstif, just use
the configure switch --with-gui=none. Also make sure you have an FTP-server
running, I always used vsftpd. To run the test I just called tiny-bench,
in it you will find how you can start it. You can also run without FTP-server
but I do not know if the problems are reproduceable.

> It would be interesting to see which new feature introduced by the
> ext4 patch queue --- probably dellayed allocation or mballoc --- is
> responsible for the slowdown.  One or the other (or both) can be
> disabled by mounting the filesystem (using a kernel with the ext4
> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>
> If it turns out that nomballoc restores the speed for afdbench, for
> example, then it will tell us where we need to look more closely.
> Ideally we would not want to have one mount option needed to optimize
> filesystem operations for large amoutns of modifications to small
> files, and another mode of operation when mostly writing to large
> files.  So if you could do a round of tests using the ext4 patch queue
> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
> improve things, try "-O nomballoc,nodelalloc" and see if you get back
> to the pre-ext4 patch queue speed), it would be very much appreciated.
>
Yes, I will try and redo the test as suggested, it might just take a while
since I just made my testing system operational.

What worries me more is the truncation of files, it makes the filesystem
unusable since you loose data. I hope there will be a solution for this.

Holger

Re: Performance of ext4

[Holger Kiehl]

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>
>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>
> Can you remind me where afdbench can be downloaded?  And if I remember
> correctly, it creates and deletes large numbers of small files,
> correct?
>
> It would be interesting to see which new feature introduced by the
> ext4 patch queue --- probably dellayed allocation or mballoc --- is
> responsible for the slowdown.  One or the other (or both) can be
> disabled by mounting the filesystem (using a kernel with the ext4
> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>
> If it turns out that nomballoc restores the speed for afdbench, for
> example, then it will tell us where we need to look more closely.
> Ideally we would not want to have one mount option needed to optimize
> filesystem operations for large amoutns of modifications to small
> files, and another mode of operation when mostly writing to large
> files.  So if you could do a round of tests using the ext4 patch queue
> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
> improve things, try "-O nomballoc,nodelalloc" and see if you get back
> to the pre-ext4 patch queue speed), it would be very much appreciated.
>
Here the results:
                                      +---------+------------+
                                      | afdbench|   bonnie++ |
                                      +---------+--------+---+
                                      |file rate| block w|%CP|
-------------------------------------+---------+--------+---+
ext3                                 | 5536.79 | 212350 | 92|
ext4-patch-queue                     | 5054.86 | 244384 | 50|
ext4-patch-queue-nodelalloc          | 4943.78 | 225819 | 92|
ext4-patch-queue-nomballoc           | 3123.49 | 244535 | 52|
ext4-patch-queue-nomballoc-nodelalloc| 4931.09 | 231332 | 91|
-------------------------------------+---------+--------+---+

Test where done with 2.6.26-rc8 and
ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928.tar.gz,
e2fsprogs is from git (27th April 2008). ext4 filesystem was created
with 'mke2fs -m 0 -t ext4dev /dev/md7' and ext3 'mke2fs -m 0 -j /dev/md7'.
Common mount options are: noatime,nodiratime,commit=15

Looking at the afdbench results I also notice that when I just take
the FTP results the results look as follows:

ext3                                 : 3465.50
ext4-patch-queue                     : 2785.58
ext4-patch-queue-nodelalloc          : 2677.39
ext4-patch-queue-nomballoc           :  219.12
ext4-patch-queue-nomballoc-nodelalloc: 2566.24

Now one can see the drop with ext4-patch-queue much clearer. The testing
of afdbench is done in two parts, one where we just link lots of small
files locally and the same test is then repeated using a network protocol
in this case FTP. So the difference is that for the filesystem lots
of new files get created. Further testing showed that when I increase
the number FTP process performance decreases in all cases but much more
for ext4-patch-queue (nearly 50% drop against ext3) as the following results
show:

ext3                                 : 2352.89
ext4-patch-queue                     : 1226.55
ext4-patch-queue-nodelalloc          : 1340.80
ext4-patch-queue-nomballoc-nodelalloc: 1181.12

I did not do the ext4-patch-queue-nomballoc test since there is obviously
something wrong here when you look at the numbers above (219.12 fps).
During that test I notice that when you try to open an existing file
with vi it can take several minutes before it opens this file. The strange
thing is that vi was not in D-state but it could not be killed, even root
could not kill it with -9.

There is also some corruption in filesystem during the test with
ext4-patch-queue and ext4-patch-queue-nomballoc. It happens when after
the test I umount the test filesystem and then mount it again the
following message appears:

root@athena:~# umount /home
root@athena:~# mount /home
mount: wrong fs type, bad option, bad superblock on /dev/md7,
        missing codepage or helper program, or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so

EXT4-fs: ext4_check_descriptors: Inode bitmap for group 256 not in group (block 117835012)!<3>EXT4-fs: group descriptors corrupted!

Using fsck this problem could be corrected. Now that one does not think I
did those test on a corrupted file system. The filesystem was newly created
for each of the above five test runs.

Regards,
Holger

Re: Performance of ext4

[Holger Kiehl]

On Mon, 7 Jul 2008, Holger Kiehl wrote:

> On Thu, 19 Jun 2008, Theodore Tso wrote:
>
>> On Wed, Jun 18, 2008 at 05:58:00AM +0000, Holger Kiehl wrote:
>>> For afdbench: 5336.41 files per second 15.63 MiB/s
>>> 
>>> So it seems that for afdbench the ext4-patch-queue is a slowdown.
>> 
>> Can you remind me where afdbench can be downloaded?  And if I remember
>> correctly, it creates and deletes large numbers of small files,
>> correct?
>> 
>> It would be interesting to see which new feature introduced by the
>> ext4 patch queue --- probably dellayed allocation or mballoc --- is
>> responsible for the slowdown.  One or the other (or both) can be
>> disabled by mounting the filesystem (using a kernel with the ext4
>> patch queue) with the mount options -O nomballoc or -O nodelalloc.
>> 
>> If it turns out that nomballoc restores the speed for afdbench, for
>> example, then it will tell us where we need to look more closely.
>> Ideally we would not want to have one mount option needed to optimize
>> filesystem operations for large amoutns of modifications to small
>> files, and another mode of operation when mostly writing to large
>> files.  So if you could do a round of tests using the ext4 patch queue
>> kernel, with -O nomballoc and -O nodelalloc (and if both seem to
>> improve things, try "-O nomballoc,nodelalloc" and see if you get back
>> to the pre-ext4 patch queue speed), it would be very much appreciated.
>> 
> Here the results:
>                                     +---------+------------+
>                                     | afdbench|   bonnie++ |
>                                     +---------+--------+---+
>                                     |file rate| block w|%CP|
> -------------------------------------+---------+--------+---+
> ext3                                 | 5536.79 | 212350 | 92|
> ext4-patch-queue                     | 5054.86 | 244384 | 50|
> ext4-patch-queue-nodelalloc          | 4943.78 | 225819 | 92|
> ext4-patch-queue-nomballoc           | 3123.49 | 244535 | 52|
> ext4-patch-queue-nomballoc-nodelalloc| 4931.09 | 231332 | 91|
> -------------------------------------+---------+--------+---+
>
> Test where done with 2.6.26-rc8 and
> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928.tar.gz,
> e2fsprogs is from git (27th April 2008). ext4 filesystem was created
> with 'mke2fs -m 0 -t ext4dev /dev/md7' and ext3 'mke2fs -m 0 -j /dev/md7'.
> Common mount options are: noatime,nodiratime,commit=15
>
> Looking at the afdbench results I also notice that when I just take
> the FTP results the results look as follows:
>
> ext3                                 : 3465.50
> ext4-patch-queue                     : 2785.58
> ext4-patch-queue-nodelalloc          : 2677.39
> ext4-patch-queue-nomballoc           :  219.12
> ext4-patch-queue-nomballoc-nodelalloc: 2566.24
>
> Now one can see the drop with ext4-patch-queue much clearer. The testing
> of afdbench is done in two parts, one where we just link lots of small
> files locally and the same test is then repeated using a network protocol
> in this case FTP. So the difference is that for the filesystem lots
> of new files get created. Further testing showed that when I increase
> the number FTP process performance decreases in all cases but much more
> for ext4-patch-queue (nearly 50% drop against ext3) as the following results
> show:
>
> ext3                                 : 2352.89
> ext4-patch-queue                     : 1226.55
> ext4-patch-queue-nodelalloc          : 1340.80
> ext4-patch-queue-nomballoc-nodelalloc: 1181.12
>
> I did not do the ext4-patch-queue-nomballoc test since there is obviously
> something wrong here when you look at the numbers above (219.12 fps).
> During that test I notice that when you try to open an existing file
> with vi it can take several minutes before it opens this file. The strange
> thing is that vi was not in D-state but it could not be killed, even root
> could not kill it with -9.
>
> There is also some corruption in filesystem during the test with
> ext4-patch-queue and ext4-patch-queue-nomballoc. It happens when after
> the test I umount the test filesystem and then mount it again the
> following message appears:
>
> root@athena:~# umount /home
> root@athena:~# mount /home
> mount: wrong fs type, bad option, bad superblock on /dev/md7,
>       missing codepage or helper program, or other error
>       In some cases useful info is found in syslog - try
>       dmesg | tail  or so
>
> EXT4-fs: ext4_check_descriptors: Inode bitmap for group 256 not in group 
> (block 117835012)!<3>EXT4-fs: group descriptors corrupted!
>
> Using fsck this problem could be corrected. Now that one does not think I
> did those test on a corrupted file system. The filesystem was newly created
> for each of the above five test runs.
>
Any ideas what I can do to help find why performance under load is nearly
halved and the group descriptor corruption?

I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
from today, but I immediatly hit an oops as soon as I untar a file, see below.

Thanks,
Holger


kjournald2 starting.  Commit interval 15 seconds
EXT4 FS on md7, internal journal
EXT4-fs: mounted filesystem with ordered data mode.
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
------------[ cut here ]------------
kernel BUG at fs/ext4/extents.c:1817!
invalid opcode: 0000 [1] SMP 
CPU 0 
Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc floppy i2c_amd756 i2c_core k8temp ohci_hcd sg button usbcore
Pid: 2757, comm: tar Not tainted 2.6.26-rc9 #1
RIP: 0010:[<ffffffff802e2722>]  [<ffffffff802e2722>] ext4_ext_get_blocks+0x9eb/0xde1
RSP: 0018:ffff81007a0f99b8  EFLAGS: 00010246
RAX: 0000000000000001 RBX: ffff81002cfd69c0 RCX: ffff81002cfd69a8
RDX: ffff81007f48c6fc RSI: 00000000ffffffff RDI: ffff81002cfd69c0
RBP: ffff81007a0f9b88 R08: ffff81007f48c6fc R09: 0000000000000000
R10: 000000000000a855 R11: 0000000000000000 R12: ffff81007f48c7b0
R13: 0000000000000001 R14: ffff81007f48c7b0 R15: 0000000000000001
FS:  00007f66afd3b780(0000) GS:ffffffff80570000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000081d000 CR3: 00000001e9e86000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process tar (pid: 2757, threadinfo ffff81007a0f8000, task ffff81007d9110e0)
Stack:  ffff81007d36c300 000000007f46e030 ffff81007a0f9b88 0000000000000001
  000000012c815bc0 ffff81007f46e030 ffff81002cfd69c0 000000007d36c300
  ffff81007a0f9bb8 ffff81007f48c6f0 000000007a0f9bc8 ffff81007f46e030
Call Trace:
  [<ffffffff802d2aaa>] ? ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff80223c42>] ? __wake_up+0x38/0x4f
  [<ffffffff802d4e0b>] ? ext4_get_blocks_wrap+0x70/0x165
  [<ffffffff8031af55>] ? __up_read+0x13/0x8a
  [<ffffffff802d5280>] ? ext4_getblk+0x62/0x170
  [<ffffffff802d7801>] ? add_dirent_to_buf+0xcb/0x2ec
  [<ffffffff802d539b>] ? ext4_bread+0xd/0x5f
  [<ffffffff802d7206>] ? ext4_append+0x3a/0x88
  [<ffffffff802d8042>] ? ext4_add_entry+0x620/0x87f
  [<ffffffff802d12ce>] ? ext4_new_inode+0xc4e/0xc78
  [<ffffffff802f58f3>] ? start_this_handle+0x2c7/0x370
  [<ffffffff802d8916>] ? ext4_add_nondir+0x18/0x4e
  [<ffffffff802d8ff8>] ? ext4_create+0xc2/0x105
  [<ffffffff802d9288>] ? ext4_lookup+0x97/0xc1
  [<ffffffff802823d6>] ? vfs_create+0x75/0xba
  [<ffffffff80284e5d>] ? do_filp_open+0x1e4/0x7f6
  [<ffffffff80279e7e>] ? sys_chown+0x5c/0x6b
  [<ffffffff80279684>] ? do_sys_open+0x46/0xca
  [<ffffffff8020b16b>] ? system_call_after_swapgs+0x7b/0x80


Code: 39 44 24 24 72 2f 66 81 fa 00 80 0f b7 c2 76 05 2d 00 80 00 00 48 8b 7c 24 30 01 f0 89 44 24 24 e8 71 d3 ff ff 3b 44 24 24 75 04 <0f> 0b eb fe 2b 44 24 24 eb 11 0f 0b eb fe c7 44 24 24 00 00 00 
RIP  [<ffffffff802e2722>] ext4_ext_get_blocks+0x9eb/0xde1
  RSP <ffff81007a0f99b8>
---[ end trace e595ecd19e9f2f92 ]---

Re: Performance of ext4

[Aneesh Kumar K.V]

On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> On Mon, 7 Jul 2008, Holger Kiehl wrote:
>
>
> I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> from today, but I immediatly hit an oops as soon as I untar a file, see below.
>

I am able to reproduce this.  Revert the patch Fix-EXT_MAX_BLOCK.patch. 

-aneesh

Revert Fix-EXT_MAX_BLOCK.patch

[Aneesh Kumar K.V]

Sending again with different subject.

On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> >
> >
> > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> >
> 
> I am able to reproduce this.  Revert the patch Fix-EXT_MAX_BLOCK.patch. 
> 
> -aneesh

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Theodore Tso]

Aneesh, for the record, how were you able to reproduce the problem?

Do you have Holger's afdbench setup, or did you use some other workload?

Thanks, regards,

						- Ted

On Thu, Jul 10, 2008 at 02:56:35PM +0530, Aneesh Kumar K.V wrote:
> Sending again with different subject.
> 
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> > On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> > >
> > >
> > > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> > >
> > 
> > I am able to reproduce this.  Revert the patch Fix-EXT_MAX_BLOCK.patch. 
> > 
> > -aneesh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Aneesh Kumar K.V]

On Thu, Jul 10, 2008 at 08:22:42AM -0400, Theodore Tso wrote:
> Aneesh, for the record, how were you able to reproduce the problem?
> 
> Do you have Holger's afdbench setup, or did you use some other workload?
> 

cp -ax /usr . 

that will cause the BUG

-aneesh

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Theodore Tso]

On Thu, Jul 10, 2008 at 06:08:03PM +0530, Aneesh Kumar K.V wrote:
> On Thu, Jul 10, 2008 at 08:22:42AM -0400, Theodore Tso wrote:
> > Aneesh, for the record, how were you able to reproduce the problem?
> > 
> > Do you have Holger's afdbench setup, or did you use some other workload?
> > 
> 
> cp -ax /usr . 
> 
> that will cause the BUG

Hmm, I wasn't able to trigger it so easily, at least, while trying to
replicate Gary Hawco's, bug.  (Which also had Fix-EXT_MAX_BLOCK
applied.)  I wonder if he wasn't able to trigger it while doing the
partial rollback, and so ext4-fix-mb_find_next_bit-return.patch was
unfairly blamed for the BUG.

In any case, both patches have been pulled out of the patch series ,
and I'll release a new patchset until we can figure out what happened.

(Gary, this is why its important to get the kernel BUG/oops message;
it helps us correlate your bug reports with others.)

						- Ted

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Theodore Tso]

Ive commented out Fix-EXT_MAX_BLOCK.patch from the series file until
we know more.

						- Ted

On Thu, Jul 10, 2008 at 02:56:35PM +0530, Aneesh Kumar K.V wrote:
> Sending again with different subject.
> 
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
> > On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
> > > On Mon, 7 Jul 2008, Holger Kiehl wrote:
> > >
> > >
> > > I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
> > > from today, but I immediatly hit an oops as soon as I untar a file, see below.
> > >
> > 
> > I am able to reproduce this.  Revert the patch Fix-EXT_MAX_BLOCK.patch. 
> > 
> > -aneesh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Holger Kiehl]

On Thu, 10 Jul 2008, Aneesh Kumar K.V wrote:

> Sending again with different subject.
>
> On Thu, Jul 10, 2008 at 02:54:25PM +0530, Aneesh Kumar K.V wrote:
>> On Thu, Jul 10, 2008 at 08:11:15AM +0000, Holger Kiehl wrote:
>>> On Mon, 7 Jul 2008, Holger Kiehl wrote:
>>>
>>>
>>> I did try newer patch queue (ext4-patch-queue-a5d48915447f44c3af6ce8e1c91d45b452977fcf)
>>> from today, but I immediatly hit an oops as soon as I untar a file, see below.
>>>
>>
>> I am able to reproduce this.  Revert the patch Fix-EXT_MAX_BLOCK.patch.
>>
Thanks. Reverting that patch also fixed it for me. I was able to do my
test however performance is down another 10% (compared to
ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
slower and slower :(

Also the group descriptors still get corrupted.

Holger

PS: http://repo.or.cz/w/ext4-patch-queue.git is empty, is that correct?

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Theodore Tso]

On Fri, Jul 11, 2008 at 09:57:47AM +0000, Holger Kiehl wrote:
> Thanks. Reverting that patch also fixed it for me. I was able to do my
> test however performance is down another 10% (compared to
> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
> slower and slower :(

How reproducible are your results?  That is, if you run the benchmarks
multiple times, how much variance is there between different runs?

If you are willing, this would be helpful.  In your ext4 patch
repository, try out commit 179a876b.  (You can do this via 
"git checkout -b rc9-rebase 179a876b"; after doing the test you can
switch the working directory of the ext4 patch queue back to the master
branch via "git checkout master".)   This commit is pretty much 
identical to your previous 52c8a02a test, modulo rebasing to -rc9.

If you see the same results, you could try going to the next patch,
via "git checkout -b i-blocks-stat ef019f0a" which also has the fix so
that stat returns a valid i_blocks field for files that have been
freshly written when delayed allocation is enabled.  Both of these
revisions rae before the patches that were causing corrupion were
added to the patch queue, so it should be fine.

The funny thing is looking at the various recent patches, I don't see
how they could be affecting performance of your patches so
significantly.  I gather afdbench is very metadata intensive, with
lots of small files, but even so, none of these patches should make
that kind of difference.  So that's why I'm wondering how much
variance there is between runs of afdbench, and whether that might be
a possible explanation.

> Also the group descriptors still get corrupted.

Hmm, can you send me the output of dumpe2fs before and after the
benchmark run which corrupts the group descriptors?  And can you send
me the output of "e2fsck -fy /dev/XXXXX >& /tmp/log", so I can see
what got corrupted?

I also note that you are using a fairly old e2fsprogs from April 27th.
You might want to try going to the just-released e2fsprogs 1.41.0
released yesterday, as that has some flex_bg layout changes that might
help out performance for afdbench.  Also note that with both the April
27th and the latest e2fsprogs 1.41.0 release, there is a mke2fs.conf
file in misc/mke2fs.conf that should be installed in /etc/mke2fs.conf
for best results.

> PS: http://repo.or.cz/w/ext4-patch-queue.git is empty, is that correct?

Nope, we're working on that.  Things seem to have gotten corrupted on
repo.or.cz, as you may have seen on another e-mail thread.  I have
established an git repository here:

	    git://git.kernel.org/pub/scm/fs/ext2/ext4-patch-queue.git

As an interim replacement.

						- Ted

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Holger Kiehl]

On Fri, 11 Jul 2008, Theodore Tso wrote:

> On Fri, Jul 11, 2008 at 09:57:47AM +0000, Holger Kiehl wrote:
>> Thanks. Reverting that patch also fixed it for me. I was able to do my
>> test however performance is down another 10% (compared to
>> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
>> slower and slower :(
>
> How reproducible are your results?  That is, if you run the benchmarks
> multiple times, how much variance is there between different runs?
>
I always run the benchmark 3 times and then take the average. Additionally
there are two main types of test one where files are moved locally (FILE)
and another where files are send via FTP. So its 6 runs with the following
results:

     FTP        FILE
     2548.81    6569.68
     2613.05    6480.86
     2599.09    6573.62

I then took all six runs added them and diveded by six giving 4564.19 fps.
Those results where achived with a5d48915 and e2fsprogs-1.41-WIP-0707.tar.bz2.

The same was done with 52c8a02a only here I used the April 27th e2fsprogs.
There I got 5054.86 fps.

Each run takes 30 minutes and approx. 10 minutes to delete the test files
from a previous run and setup the new test files, that is approx. 4 hours
for all 6 runs. I then also always do a 2 hour test run with a lot more
files and process sending files, one for FTP and one for FILE. But I
did not mention those numbers because I always did it once. But here too
one could see the numbers going down by approx. 10%.

> If you are willing, this would be helpful.  In your ext4 patch
> repository, try out commit 179a876b.  (You can do this via
> "git checkout -b rc9-rebase 179a876b"; after doing the test you can
> switch the working directory of the ext4 patch queue back to the master
> branch via "git checkout master".)   This commit is pretty much
> identical to your previous 52c8a02a test, modulo rebasing to -rc9.
>
That is why I did another test run with ext3 which I did not mention, sorry.
Here the results:

                  ext3       ext4-patch-queue
     52c8a02a     5536.79    5054.86
     a5d48915     5587.78    4564.19

So the result of ext3 are the same while ext4-patch-queue dropped the
nearly 10%.

> If you see the same results, you could try going to the next patch,
> via "git checkout -b i-blocks-stat ef019f0a" which also has the fix so
> that stat returns a valid i_blocks field for files that have been
> freshly written when delayed allocation is enabled.  Both of these
> revisions rae before the patches that were causing corrupion were
> added to the patch queue, so it should be fine.
>
> The funny thing is looking at the various recent patches, I don't see
> how they could be affecting performance of your patches so
> significantly.  I gather afdbench is very metadata intensive, with
> lots of small files, but even so, none of these patches should make
> that kind of difference.  So that's why I'm wondering how much
> variance there is between runs of afdbench, and whether that might be
> a possible explanation.
>
>> Also the group descriptors still get corrupted.
>
> Hmm, can you send me the output of dumpe2fs before and after the
> benchmark run which corrupts the group descriptors?  And can you send
> me the output of "e2fsck -fy /dev/XXXXX >& /tmp/log", so I can see
> what got corrupted?
>
> I also note that you are using a fairly old e2fsprogs from April 27th.
> You might want to try going to the just-released e2fsprogs 1.41.0
> released yesterday, as that has some flex_bg layout changes that might
> help out performance for afdbench.
>
Where those changes already in e2fsprogs-1.41-WIP-0707.tar.bz2 release?

> Also note that with both the April
> 27th and the latest e2fsprogs 1.41.0 release, there is a mke2fs.conf
> file in misc/mke2fs.conf that should be installed in /etc/mke2fs.conf
> for best results.
>
Since I made my own src rpm, I did use the misc/mke2fs.conf in both cases.
Just checked this.

Holger

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Holger Kiehl]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2485 bytes --]

On Fri, 11 Jul 2008, Theodore Tso wrote:

> On Fri, Jul 11, 2008 at 09:57:47AM +0000, Holger Kiehl wrote:
>> Thanks. Reverting that patch also fixed it for me. I was able to do my
>> test however performance is down another 10% (compared to
>> ext4-patch-queue-52c8a02a8a7b7e5915b9301e9c171b4faf22b928). ext4 is getting
>> slower and slower :(
>
> How reproducible are your results?  That is, if you run the benchmarks
> multiple times, how much variance is there between different runs?
>
> If you are willing, this would be helpful.  In your ext4 patch
> repository, try out commit 179a876b.  (You can do this via
> "git checkout -b rc9-rebase 179a876b"; after doing the test you can
> switch the working directory of the ext4 patch queue back to the master
> branch via "git checkout master".)   This commit is pretty much
> identical to your previous 52c8a02a test, modulo rebasing to -rc9.
>
> If you see the same results, you could try going to the next patch,
> via "git checkout -b i-blocks-stat ef019f0a" which also has the fix so
> that stat returns a valid i_blocks field for files that have been
> freshly written when delayed allocation is enabled.  Both of these
> revisions rae before the patches that were causing corrupion were
> added to the patch queue, so it should be fine.
>
> The funny thing is looking at the various recent patches, I don't see
> how they could be affecting performance of your patches so
> significantly.  I gather afdbench is very metadata intensive, with
> lots of small files, but even so, none of these patches should make
> that kind of difference.  So that's why I'm wondering how much
> variance there is between runs of afdbench, and whether that might be
> a possible explanation.
>
You are right. I did compare the .config of both and noticed that
CONFIG_FAIR_GROUP_SCHED was set in the rc9 test but not in rc8 test.
Doing the test without CONFIG_FAIR_GROUP_SCHED gave me back 6%.
Sorry.

>> Also the group descriptors still get corrupted.
>
> Hmm, can you send me the output of dumpe2fs before and after the
> benchmark run which corrupts the group descriptors?  And can you send
> me the output of "e2fsck -fy /dev/XXXXX >& /tmp/log", so I can see
> what got corrupted?
>
I did several test runs with dumpe2fs before and after, but it would then
not cause any corruption. Leaving it away I got the corruption. So
attached you will find dumpe2fs after corruption in unmounted state
and the e2fsck output. Is this of any use?

Holger

[-- Attachment #2: Type: APPLICATION/x-bzip2, Size: 50897 bytes --]

[-- Attachment #3: Type: APPLICATION/x-bzip2, Size: 959 bytes --]

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Theodore Tso]

On Mon, Jul 14, 2008 at 07:55:10PM +0000, Holger Kiehl wrote:
> You are right. I did compare the .config of both and noticed that
> CONFIG_FAIR_GROUP_SCHED was set in the rc9 test but not in rc8 test.
> Doing the test without CONFIG_FAIR_GROUP_SCHED gave me back 6%.
> Sorry.

You may have told us already, but can you tell me the full
configuration of your benchmark machine?  (i.e., how many CPU's, how
much memory, etc.)  Also, what are the current mount options you are
currently using?  And have you redone the ext3 benchmark number with
barriers enabled?  Or was that the original number done with default
mount options that leave barriers disabled?

Thanks!!

						- Ted

Re: Revert Fix-EXT_MAX_BLOCK.patch

[Holger Kiehl]

On Mon, 14 Jul 2008, Theodore Tso wrote:

> On Mon, Jul 14, 2008 at 07:55:10PM +0000, Holger Kiehl wrote:
>> You are right. I did compare the .config of both and noticed that
>> CONFIG_FAIR_GROUP_SCHED was set in the rc9 test but not in rc8 test.
>> Doing the test without CONFIG_FAIR_GROUP_SCHED gave me back 6%.
>> Sorry.
>
> You may have told us already, but can you tell me the full
> configuration of your benchmark machine?  (i.e., how many CPU's, how
> much memory, etc.)  Also, what are the current mount options you are
> currently using?  And have you redone the ext3 benchmark number with
> barriers enabled?  Or was that the original number done with default
> mount options that leave barriers disabled?
>
System has 4 Opteron 848 (2200MHz) CPU's, 8GB DDR400 memory (2GB per CPU),
8 15K SCSI U320 disk connected via two dual Adaptec ASC-39320A controllers
used as data disk, system disk is on 2 sata disks. The eight data disks,
where I do the testing, are combined to a software raid 1+0 (not Raid 10).
Because I am using software raid barriers are disabled, so for all testing
barriers where disabled. The mount options are: noatime,nodiratime,commit=15

There are also a 4 port and a 2 port Gigabit network card, but these where
not used. Here the output of lspci:

00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:1a.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:1a.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:1a.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:1a.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:1b.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:1b.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:1b.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:1b.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
01:04.0 Mass storage controller: Promise Technology, Inc. PDC40775 (SATA 300 TX2plus) (rev 02)
01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
02:02.0 SCSI storage controller: Adaptec ASC-39320A U320 (rev 10)
02:02.1 SCSI storage controller: Adaptec ASC-39320A U320 (rev 10)
02:03.0 SCSI storage controller: Adaptec ASC-39320A U320 (rev 10)
02:03.1 SCSI storage controller: Adaptec ASC-39320A U320 (rev 10)
03:01.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
03:03.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
03:03.1 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
04:04.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (rev 01)
04:04.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (rev 01)
04:06.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (rev 01)
04:06.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (rev 01)

Holger

Re: Performance of ext4

[Theodore Tso]

On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
> Note how the size of file results.24033.helena.dwd.de changes from
> 9230 before the test to 8208 bytes after the test. Also note the
> date both have the same timestamp "2008-06-17 04:35". I have made a
> copy of results.24033.helena.dwd.de before the test and compared it
> with that after the test. The file is just truncated by 1022 bytes
> and there is no garbage.

So the corruption is always a truncation, correct?

Did you notice the problem with ext4 w/o the patch queue?  I have a
suspicion that the problem may have been introduced by the delayed
allocation code, but I don't have hard evidence.  When you rerun your
benchmark (which seems to be the closest thing we have to a
reproduction case), it would be interesting to know if the problem
goes away with -o nodelalloc (again, it would localize where we need
to look).

Thanks, regards,

						- Ted

Re: Performance of ext4

[Eric Sandeen]

Theodore Tso wrote:
> On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
>> Note how the size of file results.24033.helena.dwd.de changes from
>> 9230 before the test to 8208 bytes after the test. Also note the
>> date both have the same timestamp "2008-06-17 04:35". I have made a
>> copy of results.24033.helena.dwd.de before the test and compared it
>> with that after the test. The file is just truncated by 1022 bytes
>> and there is no garbage.
> 
> So the corruption is always a truncation, correct?
> 
> Did you notice the problem with ext4 w/o the patch queue?  I have a
> suspicion that the problem may have been introduced by the delayed
> allocation code, but I don't have hard evidence.  When you rerun your
> benchmark (which seems to be the closest thing we have to a
> reproduction case), it would be interesting to know if the problem
> goes away with -o nodelalloc (again, it would localize where we need
> to look).
> 
> Thanks, regards,

It might be worth runninga "simple" fsx under your kernel too; last time
I tested fsx it was still happy and it exercises fs ops (including
truncate) at random...

-Eric

Re: Performance of ext4

[Theodore Tso]

On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
> 
> It might be worth runninga "simple" fsx under your kernel too; last time
> I tested fsx it was still happy and it exercises fs ops (including
> truncate) at random...
> 

>From what Holger described, it's doubtful that the bug is in the
truncate operation.  It sounds like i_size is actually dropping in
size at some pointer long after the file was written.  If I had to
guess the value in the inode cache is correct; and perhaps so is the
value on the journal.  But somehow, the wrong value is getting written
to disk (remember the jbd layer can keep up to three different
versions of filesystem metadata in memory, because most of the time we
don't block modifications to the filesystem while we are in the middle
of writing a previous commit to disk).  So depending on whether the
inode gets redirtied or not, the inconsistency could self-heal, and if
the inode never gets pushed out of memory due to memory pressure, the
problem might not be noticed until the system reboots or the
filesystem is unmounted.

This is one of the reasons why I'm a bit suspicious that the problem
may lie in the delayed allocation code; changing i_size without first
starting a transaction could lead to this sort of problem, for
example, and the delayed allocation could represent a different code
path where file blocks get allocated and i_size gets changed.

     	   	       	   	     	 	- Ted

Re: Performance of ext4

[Mingming]

On Thu, 2008-06-19 at 13:42 -0400, Theodore Tso wrote:
> On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
> > 
> > It might be worth runninga "simple" fsx under your kernel too; last time
> > I tested fsx it was still happy and it exercises fs ops (including
> > truncate) at random...
> > 
> 
> From what Holger described, it's doubtful that the bug is in the
> truncate operation.  It sounds like i_size is actually dropping in
> size at some pointer long after the file was written.  If I had to
> guess the value in the inode cache is correct; and perhaps so is the
> value on the journal.  But somehow, the wrong value is getting written
> to disk (remember the jbd layer can keep up to three different
> versions of filesystem metadata in memory, because most of the time we
> don't block modifications to the filesystem while we are in the middle
> of writing a previous commit to disk).  So depending on whether the
> inode gets redirtied or not, the inconsistency could self-heal, and if
> the inode never gets pushed out of memory due to memory pressure, the
> problem might not be noticed until the system reboots or the
> filesystem is unmounted.
> 
> This is one of the reasons why I'm a bit suspicious that the problem
> may lie in the delayed allocation code; changing i_size without first
> starting a transaction could lead to this sort of problem, for
> example, and the delayed allocation could represent a different code
> path where file blocks get allocated and i_size gets changed.
> 

I tend to agree.  Without delayed allocation, the in-memory i_size and
on-disk i_disksize normally match each other, since we do block
allocation at prepare_write/write_begin time, and the i_size update just
immedietly around that time. However, with delayed allocation, the in
memory i_size is being update around prepare_write/commit_write, but the
i_disksize won't updated until later writepage/writepages() time. The
window now gets much larger.

With writeback mode, since there is no ordering there, I think it's
possible the the inode dirty pages have been sync to disk and the inode
structure being pushed out of the memory due to memory pressure, before
the i_disksize update cached in jbd2 reach to disk. Perhaps that explain
the "truncation"?

Not sure if this still a issue with the delalloc on new ordered mode, I
guess as long as the inode is on jinode list, and that inode can't push
out of memeory due to memory pressure since jbd is referencing it, then
this seems couldn't happen...



>      	   	       	   	     	 	- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance of ext4

[Holger Kiehl]

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Thu, Jun 19, 2008 at 11:41:17AM -0500, Eric Sandeen wrote:
>>
>> It might be worth runninga "simple" fsx under your kernel too; last time
>> I tested fsx it was still happy and it exercises fs ops (including
>> truncate) at random...
>>
>
> From what Holger described, it's doubtful that the bug is in the
> truncate operation.
>
Correct, the benchmark just copies, moves, hardlinks and deletes a lot
of small files. It also overwrites existing files but not at the same
scale it does the other operations.

> It sounds like i_size is actually dropping in
> size at some pointer long after the file was written.  If I had to
> guess the value in the inode cache is correct; and perhaps so is the
> value on the journal.  But somehow, the wrong value is getting written
> to disk (remember the jbd layer can keep up to three different
> versions of filesystem metadata in memory, because most of the time we
> don't block modifications to the filesystem while we are in the middle
> of writing a previous commit to disk).  So depending on whether the
> inode gets redirtied or not, the inconsistency could self-heal, and if
> the inode never gets pushed out of memory due to memory pressure, the
> problem might not be noticed until the system reboots or the
> filesystem is unmounted.
>
I always had the feeling that waiting a day or unmounting caused a lot
more truncation. On my system at home for example I mounted the test
filesystem again and saw that files where truncated and I am pretty sure
that when I looked at those files during and shortly after the test they
where still complete. But I will recheck and do test as you suggested.

What I find strange is that the missing parts of the file are not for
example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
of bytes.

Holger

Re: Performance of ext4

[Theodore Tso]

On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>> It sounds like i_size is actually dropping in
>> size at some pointer long after the file was written.  If I had to

sorry, "at some point"...
            
>> guess the value in the inode cache is correct; and perhaps so is the
>> value on the journal.  But somehow, the wrong value is getting written
>> to disk 

Or, "the right value is never getting written to disk".  (Which as I
think about it is more likely; it's likely that an update to i_size is
getting *lost*, perhaps because the delalloc code is possibly
modifying i_size without starting a transaction first.  Again this is
just a guess.)

> What I find strange is that the missing parts of the file are not for
> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
> of bytes.

Is there any chance the truncation point is related to how the program
is writing its output file?  i.e., if it is a text file, is the
truncation happening after a new-line or when the stdio library might
have done an explicit or implicit fflush()?

							- Ted

Re: Performance of ext4

[Holger Kiehl]

On Fri, 20 Jun 2008, Theodore Tso wrote:

> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>> It sounds like i_size is actually dropping in
>>> size at some pointer long after the file was written.  If I had to
>
> sorry, "at some point"...
>
>>> guess the value in the inode cache is correct; and perhaps so is the
>>> value on the journal.  But somehow, the wrong value is getting written
>>> to disk
>
> Or, "the right value is never getting written to disk".  (Which as I
> think about it is more likely; it's likely that an update to i_size is
> getting *lost*, perhaps because the delalloc code is possibly
> modifying i_size without starting a transaction first.  Again this is
> just a guess.)
>
>> What I find strange is that the missing parts of the file are not for
>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>> of bytes.
>
> Is there any chance the truncation point is related to how the program
> is writing its output file?  i.e., if it is a text file, is the
> truncation happening after a new-line or when the stdio library might
> have done an explicit or implicit fflush()?
>
When the benchmark runs it writes to stdout and with tee to the result
file. It first writes some information about the system, prepares the
test files (creates lots of small files), calls sync and then starts
the test. Then every minute one line gets written to the result file.
Often I have seen that everything after the sync was missing. But
sometimes it happened that some parts at the end are missing. But it
was always a clean cut, that is there where no lines that where cut
partially. The lines where always complete.

Holger

Re: Performance of ext4

[Aneesh Kumar K.V]

On Fri, Jun 20, 2008 at 09:21:48AM +0000, Holger Kiehl wrote:
> On Fri, 20 Jun 2008, Theodore Tso wrote:
>
>> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>>> It sounds like i_size is actually dropping in
>>>> size at some pointer long after the file was written.  If I had to
>>
>> sorry, "at some point"...
>>
>>>> guess the value in the inode cache is correct; and perhaps so is the
>>>> value on the journal.  But somehow, the wrong value is getting written
>>>> to disk
>>
>> Or, "the right value is never getting written to disk".  (Which as I
>> think about it is more likely; it's likely that an update to i_size is
>> getting *lost*, perhaps because the delalloc code is possibly
>> modifying i_size without starting a transaction first.  Again this is
>> just a guess.)
>>
>>> What I find strange is that the missing parts of the file are not for
>>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>>> of bytes.
>>
>> Is there any chance the truncation point is related to how the program
>> is writing its output file?  i.e., if it is a text file, is the
>> truncation happening after a new-line or when the stdio library might
>> have done an explicit or implicit fflush()?
>>
> When the benchmark runs it writes to stdout and with tee to the result
> file. It first writes some information about the system, prepares the
> test files (creates lots of small files), calls sync and then starts
> the test. Then every minute one line gets written to the result file.
> Often I have seen that everything after the sync was missing. But
> sometimes it happened that some parts at the end are missing. But it
> was always a clean cut, that is there where no lines that where cut
> partially. The lines where always complete.
>

I found one place where we fail to update i_disksize. Can you try this
patch ?

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..9fa737f 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
 	loff_t size;
 	unsigned long len;
 	handle_t *handle = NULL;
+	ext4_lblk_t block;
+	loff_t disksize;
 	struct buffer_head *page_bufs;
+	struct buffer_head *bh, *head;
 	struct inode *inode = page->mapping->host;
 
 	handle = ext4_journal_current_handle();
@@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
 	else
 		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
 
+	 if (ret)
+		 return ret;
+	/*
+	 * When called via shrink_page_list and if we don't have any unmapped
+	 * buffer_head we still could have written some new content in an
+	 * already mapped buffer. That means we need to extent i_disksize here
+	 */
+	/* Find the last logical block number in the page. */
+	block = (sector_t)page->index << (PAGE_CACHE_SHIFT - inode->i_blkbits);
+	bh = head = page_buffers(page);
+	do {
+		bh = bh->b_this_page;
+		block++;
+	} while (bh != head);
+
+	disksize = ((loff_t) block) << inode->i_blkbits;
+	if (disksize > i_size_read(inode))
+		disksize = i_size_read(inode);
+	if (disksize > EXT4_I(inode)->i_disksize) {
+		/*
+		 * XXX: replace with spinlock if seen contended -bzzz
+		 */
+		down_write(&EXT4_I(inode)->i_data_sem);
+		if (disksize > EXT4_I(inode)->i_disksize)
+			EXT4_I(inode)->i_disksize = disksize;
+		up_write(&EXT4_I(inode)->i_data_sem);
+
+		if (EXT4_I(inode)->i_disksize == disksize) {
+			ret = ext4_mark_inode_dirty(handle, inode);
+			return ret;
+		}
+	}
 	return ret;
 }
 

Re: Performance of ext4

[Mingming]

On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:

> I found one place where we fail to update i_disksize. Can you try this
> patch ?
> 
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 33f940b..9fa737f 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
>  	loff_t size;
>  	unsigned long len;
>  	handle_t *handle = NULL;
> +	ext4_lblk_t block;
> +	loff_t disksize;
>  	struct buffer_head *page_bufs;
> +	struct buffer_head *bh, *head;
>  	struct inode *inode = page->mapping->host;
> 
>  	handle = ext4_journal_current_handle();
> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
>  	else
>  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> 
> +	 if (ret)
> +		 return ret;
> +	/*
> +	 * When called via shrink_page_list and if we don't have any unmapped
> +	 * buffer_head we still could have written some new content in an
> +	 * already mapped buffer. That means we need to extent i_disksize here
> +	 */

In this case(when extend the file without need block allocation),
wouldn't make sense to update the i_disksize at write_end() time? So
that the window of i_size different from i_disksize could be much
smaller in this case.


Something like below? (untested)

Signed-off-by: Mingming Cao <cmm@us.ibm.com>
---
 fs/ext4/inode.c |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 60 insertions(+), 6 deletions(-)

Index: linux-2.6.26-rc6/fs/ext4/inode.c
===================================================================
--- linux-2.6.26-rc6.orig/fs/ext4/inode.c	2008-06-23 17:28:01.000000000 -0700
+++ linux-2.6.26-rc6/fs/ext4/inode.c	2008-06-23 17:28:10.000000000 -0700
@@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
 	return ret;
 }
 
+static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
+{
+	return !buffer_mapped(bh) || buffer_delay(bh);
+}
+
+static int ext4_da_write_end(struct file *file,
+				struct address_space *mapping,
+				loff_t pos, unsigned len, unsigned copied,
+				struct page *page, void *fsdata)
+{
+	handle_t *handle = NULL;
+	struct inode *inode = mapping->host;
+	int needed_blocks = ext4_writepage_trans_blocks(inode);
+	unsigned from, to;
+	int ret = 0, ret2;
+
+	from = pos & (PAGE_CACHE_SIZE - 1);
+	to = from + len;
+
+	if (ret == 0) {
+		/*
+		 * generic_write_end() will run mark_inode_dirty() if i_size
+		 * changes.  So let's piggyback the i_disksize mark_inode_dirty
+		 * into that.
+		 */
+		loff_t new_i_size;
+
+		new_i_size = pos + copied;
+		if (new_i_size > EXT4_I(inode)->i_disksize)
+			if (!walk_page_buffers(NULL, page_buffers(page),
+					       0, len, NULL, ext4_bh_unmapped_or_delay)){
+				/*
+				 * Updating i_disksize when extending file without
+				 * need block allocation
+				 */
+        			handle = ext4_journal_start(inode, needed_blocks);
+				if (IS_ERR(handle)) {
+			                ret = PTR_ERR(handle);
+			                return ret;
+				}
+				if (ext4_should_order_data(inode))
+					ret = ext4_jbd2_file_inode(handle, inode);
+
+				EXT4_I(inode)->i_disksize = new_i_size;
+			}
+		ret2 = generic_write_end(file, mapping, pos, len, copied,
+							page, fsdata);
+		copied = ret2;
+		if (ret2 < 0)
+			ret = ret2;
+	}
+	if (handle)
+		ret2 = ext4_journal_stop(handle);
+	if (!ret)
+		ret = ret2;
+
+	return ret ? ret : copied;
+}
+
 static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
 {
 	struct buffer_head *head, *bh;
@@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
 	return 0;
 }
 
-static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
-{
-	return !buffer_mapped(bh) || buffer_delay(bh);
-}

Re: Performance of ext4

[Aneesh Kumar K.V]

On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> 
> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> 
> > I found one place where we fail to update i_disksize. Can you try this
> > patch ?
> > 
> > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > index 33f940b..9fa737f 100644
> > --- a/fs/ext4/inode.c
> > +++ b/fs/ext4/inode.c
> > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> >  	loff_t size;
> >  	unsigned long len;
> >  	handle_t *handle = NULL;
> > +	ext4_lblk_t block;
> > +	loff_t disksize;
> >  	struct buffer_head *page_bufs;
> > +	struct buffer_head *bh, *head;
> >  	struct inode *inode = page->mapping->host;
> > 
> >  	handle = ext4_journal_current_handle();
> > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> >  	else
> >  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > 
> > +	 if (ret)
> > +		 return ret;
> > +	/*
> > +	 * When called via shrink_page_list and if we don't have any unmapped
> > +	 * buffer_head we still could have written some new content in an
> > +	 * already mapped buffer. That means we need to extent i_disksize here
> > +	 */
> 
> In this case(when extend the file without need block allocation),
> wouldn't make sense to update the i_disksize at write_end() time? So
> that the window of i_size different from i_disksize could be much
> smaller in this case.
> 
> 
> Something like below? (untested)

In this case you will have to start a transaction in write_begin . With
the below code transaction is started inside page_lock. Also I don't
think we need needed_blocks credit just 1 should be enough because we
are not doing any block allocation here. We just need to update the
inode block.


-aneesh

> 
> Signed-off-by: Mingming Cao <cmm@us.ibm.com>
> ---
>  fs/ext4/inode.c |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
>  1 file changed, 60 insertions(+), 6 deletions(-)
> 
> Index: linux-2.6.26-rc6/fs/ext4/inode.c
> ===================================================================
> --- linux-2.6.26-rc6.orig/fs/ext4/inode.c	2008-06-23 17:28:01.000000000 -0700
> +++ linux-2.6.26-rc6/fs/ext4/inode.c	2008-06-23 17:28:10.000000000 -0700
> @@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
>  	return ret;
>  }
> 
> +static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> +{
> +	return !buffer_mapped(bh) || buffer_delay(bh);
> +}
> +
> +static int ext4_da_write_end(struct file *file,
> +				struct address_space *mapping,
> +				loff_t pos, unsigned len, unsigned copied,
> +				struct page *page, void *fsdata)
> +{
> +	handle_t *handle = NULL;
> +	struct inode *inode = mapping->host;
> +	int needed_blocks = ext4_writepage_trans_blocks(inode);
> +	unsigned from, to;
> +	int ret = 0, ret2;
> +
> +	from = pos & (PAGE_CACHE_SIZE - 1);
> +	to = from + len;
> +
> +	if (ret == 0) {
> +		/*
> +		 * generic_write_end() will run mark_inode_dirty() if i_size
> +		 * changes.  So let's piggyback the i_disksize mark_inode_dirty
> +		 * into that.
> +		 */
> +		loff_t new_i_size;
> +
> +		new_i_size = pos + copied;
> +		if (new_i_size > EXT4_I(inode)->i_disksize)
> +			if (!walk_page_buffers(NULL, page_buffers(page),
> +					       0, len, NULL, ext4_bh_unmapped_or_delay)){
> +				/*
> +				 * Updating i_disksize when extending file without
> +				 * need block allocation
> +				 */
> +        			handle = ext4_journal_start(inode, needed_blocks);
> +				if (IS_ERR(handle)) {
> +			                ret = PTR_ERR(handle);
> +			                return ret;
> +				}
> +				if (ext4_should_order_data(inode))
> +					ret = ext4_jbd2_file_inode(handle, inode);
> +
> +				EXT4_I(inode)->i_disksize = new_i_size;
> +			}
> +		ret2 = generic_write_end(file, mapping, pos, len, copied,
> +							page, fsdata);
> +		copied = ret2;
> +		if (ret2 < 0)
> +			ret = ret2;
> +	}
> +	if (handle)
> +		ret2 = ext4_journal_stop(handle);
> +	if (!ret)
> +		ret = ret2;
> +
> +	return ret ? ret : copied;
> +}
> +
>  static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
>  {
>  	struct buffer_head *head, *bh;
> @@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
>  	return 0;
>  }
> 
> -static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> -{
> -	return !buffer_mapped(bh) || buffer_delay(bh);
> -}
> -
>  /*
>   * Note that we don't need to start a transaction unless we're journaling data
>   * because we should have holes filled from ext4_page_mkwrite(). We even don't
> @@ -2050,7 +2104,7 @@ static const struct address_space_operat
>  	.writepages	= ext4_da_writepages,
>  	.sync_page	= block_sync_page,
>  	.write_begin	= ext4_da_write_begin,
> -	.write_end	= generic_write_end,
> +	.write_end	= ext4_da_write_end,
>  	.bmap		= ext4_bmap,
>  	.invalidatepage	= ext4_da_invalidatepage,
>  	.releasepage	= ext4_releasepage,
> 
> 

Re: Performance of ext4

[Aneesh Kumar K.V]

On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> > 
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> > 
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > > 
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > >  	loff_t size;
> > >  	unsigned long len;
> > >  	handle_t *handle = NULL;
> > > +	ext4_lblk_t block;
> > > +	loff_t disksize;
> > >  	struct buffer_head *page_bufs;
> > > +	struct buffer_head *bh, *head;
> > >  	struct inode *inode = page->mapping->host;
> > > 
> > >  	handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > >  	else
> > >  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > > 
> > > +	 if (ret)
> > > +		 return ret;
> > > +	/*
> > > +	 * When called via shrink_page_list and if we don't have any unmapped
> > > +	 * buffer_head we still could have written some new content in an
> > > +	 * already mapped buffer. That means we need to extent i_disksize here
> > > +	 */
> > 
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> > 
> > 
> > Something like below? (untested)
> 
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
> 
> 

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..0ccf7b9 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 	struct page *page;
 	pgoff_t index;
 	unsigned from, to;
+	handle_t *handle;
 	struct inode *inode = mapping->host;
 
 	index = pos >> PAGE_CACHE_SHIFT;
@@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 	to = from + len;
 
 retry:
+	/*
+	 * If we are writing towards the end of an already mapped
+	 * buffer_head, we don't do any block allocation. But we
+	 * need to update i_disksize.
+	 */
+	handle = ext4_journal_start(inode, 1);
+	if (IS_ERR(handle)) {
+		ret = PTR_ERR(handle);
+		goto out;
+	}
+
 	page = __grab_cache_page(mapping, index);
 	if (!page)
 		return -ENOMEM;
@@ -1786,15 +1798,65 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 							ext4_da_get_block_prep);
 	if (ret < 0) {
 		unlock_page(page);
+		ext4_journal_stop(handle);
 		page_cache_release(page);
 	}
 
 	if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
 		goto retry;
 
+out:
 	return ret;
 }
 
+static int ext4_da_write_end(struct file *file,
+				struct address_space *mapping,
+				loff_t pos, unsigned len, unsigned copied,
+				struct page *page, void *fsdata)
+{
+	handle_t *handle = ext4_journal_current_handle();
+	struct inode *inode = mapping->host;
+	unsigned from, to;
+	int ret = 0, ret2;
+
+	from = pos & (PAGE_CACHE_SIZE - 1);
+	to = from + len;
+
+	if (ret == 0) {
+		/*
+		 * generic_write_end() will run mark_inode_dirty() if i_size
+		 * changes.  So let's piggyback the i_disksize mark_inode_dirty
+		 * into that.
+		 */
+		loff_t new_i_size;
+
+		new_i_size = pos + copied;
+		if (new_i_size > EXT4_I(inode)->i_disksize)
+			if (!walk_page_buffers(NULL, page_buffers(page),
+					       0, len, NULL,
+					       ext4_bh_unmapped_or_delay)) {
+				/*
+				 * Updating i_disksize when extending file without
+				 * need block allocation
+				 */
+				if (ext4_should_order_data(inode))
+					ret = ext4_jbd2_file_inode(handle, inode);
+
+				EXT4_I(inode)->i_disksize = new_i_size;
+			}
+		ret2 = generic_write_end(file, mapping, pos,
+					len, copied, page, fsdata);
+		copied = ret2;
+		if (ret2 < 0)
+			ret = ret2;
+	}
+	ret2 = ext4_journal_stop(handle);
+	if (!ret)
+		ret = ret2;
+
+	return ret ? ret : copied;
+}
+
 static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
 {
 	/*
@@ -2250,7 +2312,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
 	.writepages	= ext4_da_writepages,
 	.sync_page	= block_sync_page,
 	.write_begin	= ext4_da_write_begin,
-	.write_end	= generic_write_end,
+	.write_end	= ext4_da_write_end,
 	.bmap		= ext4_bmap,
 	.invalidatepage	= ext4_da_invalidatepage,
 	.releasepage	= ext4_releasepage,

Re: Performance of ext4

[Aneesh Kumar K.V]

On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> > 
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> > 
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > > 
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > >  	loff_t size;
> > >  	unsigned long len;
> > >  	handle_t *handle = NULL;
> > > +	ext4_lblk_t block;
> > > +	loff_t disksize;
> > >  	struct buffer_head *page_bufs;
> > > +	struct buffer_head *bh, *head;
> > >  	struct inode *inode = page->mapping->host;
> > > 
> > >  	handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > >  	else
> > >  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > > 
> > > +	 if (ret)
> > > +		 return ret;
> > > +	/*
> > > +	 * When called via shrink_page_list and if we don't have any unmapped
> > > +	 * buffer_head we still could have written some new content in an
> > > +	 * already mapped buffer. That means we need to extent i_disksize here
> > > +	 */
> > 
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> > 
> > 
> > Something like below? (untested)
> 
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
> 
> 

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 33f940b..bc925c5 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 	struct page *page;
 	pgoff_t index;
 	unsigned from, to;
+	handle_t *handle;
 	struct inode *inode = mapping->host;
 
 	index = pos >> PAGE_CACHE_SHIFT;
@@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 	to = from + len;
 
 retry:
+	/*
+	 * If we are writing towards the end of an already mapped
+	 * buffer_head, we don't do any block allocation. But we
+	 * need to update i_disksize.
+	 */
+	handle = ext4_journal_start(inode, 1);
+	if (IS_ERR(handle)) {
+		ret = PTR_ERR(handle);
+		goto out;
+	}
+
 	page = __grab_cache_page(mapping, index);
 	if (!page)
 		return -ENOMEM;
@@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
 							ext4_da_get_block_prep);
 	if (ret < 0) {
 		unlock_page(page);
+		ext4_journal_stop(handle);
 		page_cache_release(page);
 	}
 
 	if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
 		goto retry;
 
+out:
 	return ret;
 }
 
+static int ext4_da_write_end(struct file *file,
+				struct address_space *mapping,
+				loff_t pos, unsigned len, unsigned copied,
+				struct page *page, void *fsdata)
+{
+	loff_t new_i_size;
+	unsigned from, to;
+	int ret = 0, ret2;
+	struct inode *inode = mapping->host;
+	handle_t *handle = ext4_journal_current_handle();
+
+	from = pos & (PAGE_CACHE_SIZE - 1);
+	to = from + len;
+
+	/*
+	 * generic_write_end() will run mark_inode_dirty() if i_size
+	 * changes.  So let's piggyback the i_disksize mark_inode_dirty
+	 * into that.
+	 */
+
+	new_i_size = pos + copied;
+	if (new_i_size > EXT4_I(inode)->i_disksize)
+		if (!walk_page_buffers(NULL, page_buffers(page),
+					0, len, NULL,
+					ext4_bh_unmapped_or_delay)) {
+			/*
+			 * Updating i_disksize when extending file without
+			 * need block allocation
+			 */
+			if (ext4_should_order_data(inode))
+				ret = ext4_jbd2_file_inode(handle, inode);
+
+			EXT4_I(inode)->i_disksize = new_i_size;
+		}
+	ret2 = generic_write_end(file, mapping, pos,
+			len, copied, page, fsdata);
+	copied = ret2;
+	if (ret2 < 0)
+		ret = ret2;
+	ret2 = ext4_journal_stop(handle);
+	if (!ret)
+		ret = ret2;
+
+	return ret ? ret : copied;
+}
+
 static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
 {
 	/*
@@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
 	.writepages	= ext4_da_writepages,
 	.sync_page	= block_sync_page,
 	.write_begin	= ext4_da_write_begin,
-	.write_end	= generic_write_end,
+	.write_end	= ext4_da_write_end,
 	.bmap		= ext4_bmap,
 	.invalidatepage	= ext4_da_invalidatepage,
 	.releasepage	= ext4_releasepage,

Re: Performance of ext4

[Holger Kiehl]

On Tue, 24 Jun 2008, Aneesh Kumar K.V wrote:

> On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
>> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
>>>
>>> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
>>>
>>>> I found one place where we fail to update i_disksize. Can you try this
>>>> patch ?
>>>>
>>>> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
>>>> index 33f940b..9fa737f 100644
>>>> --- a/fs/ext4/inode.c
>>>> +++ b/fs/ext4/inode.c
>>>> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
>>>>  	loff_t size;
>>>>  	unsigned long len;
>>>>  	handle_t *handle = NULL;
>>>> +	ext4_lblk_t block;
>>>> +	loff_t disksize;
>>>>  	struct buffer_head *page_bufs;
>>>> +	struct buffer_head *bh, *head;
>>>>  	struct inode *inode = page->mapping->host;
>>>>
>>>>  	handle = ext4_journal_current_handle();
>>>> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
>>>>  	else
>>>>  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
>>>>
>>>> +	 if (ret)
>>>> +		 return ret;
>>>> +	/*
>>>> +	 * When called via shrink_page_list and if we don't have any unmapped
>>>> +	 * buffer_head we still could have written some new content in an
>>>> +	 * already mapped buffer. That means we need to extent i_disksize here
>>>> +	 */
>>>
>>> In this case(when extend the file without need block allocation),
>>> wouldn't make sense to update the i_disksize at write_end() time? So
>>> that the window of i_size different from i_disksize could be much
>>> smaller in this case.
>>>
>>>
>>> Something like below? (untested)
>>
>> In this case you will have to start a transaction in write_begin . With
>> the below code transaction is started inside page_lock. Also I don't
>> think we need needed_blocks credit just 1 should be enough because we
>> are not doing any block allocation here. We just need to update the
>> inode block.
>>
>>
>
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 33f940b..bc925c5 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> 	struct page *page;
> 	pgoff_t index;
> 	unsigned from, to;
> +	handle_t *handle;
> 	struct inode *inode = mapping->host;
>
> 	index = pos >> PAGE_CACHE_SHIFT;
> @@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> 	to = from + len;
>
> retry:
> +	/*
> +	 * If we are writing towards the end of an already mapped
> +	 * buffer_head, we don't do any block allocation. But we
> +	 * need to update i_disksize.
> +	 */
> +	handle = ext4_journal_start(inode, 1);
> +	if (IS_ERR(handle)) {
> +		ret = PTR_ERR(handle);
> +		goto out;
> +	}
> +
> 	page = __grab_cache_page(mapping, index);
> 	if (!page)
> 		return -ENOMEM;
> @@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> 							ext4_da_get_block_prep);
> 	if (ret < 0) {
> 		unlock_page(page);
> +		ext4_journal_stop(handle);
> 		page_cache_release(page);
> 	}
>
> 	if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
> 		goto retry;
>
> +out:
> 	return ret;
> }
>
> +static int ext4_da_write_end(struct file *file,
> +				struct address_space *mapping,
> +				loff_t pos, unsigned len, unsigned copied,
> +				struct page *page, void *fsdata)
> +{
> +	loff_t new_i_size;
> +	unsigned from, to;
> +	int ret = 0, ret2;
> +	struct inode *inode = mapping->host;
> +	handle_t *handle = ext4_journal_current_handle();
> +
> +	from = pos & (PAGE_CACHE_SIZE - 1);
> +	to = from + len;
> +
> +	/*
> +	 * generic_write_end() will run mark_inode_dirty() if i_size
> +	 * changes.  So let's piggyback the i_disksize mark_inode_dirty
> +	 * into that.
> +	 */
> +
> +	new_i_size = pos + copied;
> +	if (new_i_size > EXT4_I(inode)->i_disksize)
> +		if (!walk_page_buffers(NULL, page_buffers(page),
> +					0, len, NULL,
> +					ext4_bh_unmapped_or_delay)) {
> +			/*
> +			 * Updating i_disksize when extending file without
> +			 * need block allocation
> +			 */
> +			if (ext4_should_order_data(inode))
> +				ret = ext4_jbd2_file_inode(handle, inode);
> +
> +			EXT4_I(inode)->i_disksize = new_i_size;
> +		}
> +	ret2 = generic_write_end(file, mapping, pos,
> +			len, copied, page, fsdata);
> +	copied = ret2;
> +	if (ret2 < 0)
> +		ret = ret2;
> +	ret2 = ext4_journal_stop(handle);
> +	if (!ret)
> +		ret = ret2;
> +
> +	return ret ? ret : copied;
> +}
> +
> static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> {
> 	/*
> @@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
> 	.writepages	= ext4_da_writepages,
> 	.sync_page	= block_sync_page,
> 	.write_begin	= ext4_da_write_begin,
> -	.write_end	= generic_write_end,
> +	.write_end	= ext4_da_write_end,
> 	.bmap		= ext4_bmap,
> 	.invalidatepage	= ext4_da_invalidatepage,
> 	.releasepage	= ext4_releasepage,
>
Yes, with this patch applied on top of latest patch queue I no longer
get truncated files, after running a short test. Tomorrow I will do some
more thorough testing and use the patch you have send to me in a separate
mail. The above patch did not apply but it was easy to apply by hand.

Thanks a lot for the patch!

Holger

Re: Performance of ext4

[Mingming]

On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> On Tue, 24 Jun 2008, Aneesh Kumar K.V wrote:
> 
> > On Tue, Jun 24, 2008 at 08:37:21AM +0530, Aneesh Kumar K.V wrote:
> >> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> >>>
> >>> On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> >>>
> >>>> I found one place where we fail to update i_disksize. Can you try this
> >>>> patch ?
> >>>>
> >>>> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> >>>> index 33f940b..9fa737f 100644
> >>>> --- a/fs/ext4/inode.c
> >>>> +++ b/fs/ext4/inode.c
> >>>> @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> >>>>  	loff_t size;
> >>>>  	unsigned long len;
> >>>>  	handle_t *handle = NULL;
> >>>> +	ext4_lblk_t block;
> >>>> +	loff_t disksize;
> >>>>  	struct buffer_head *page_bufs;
> >>>> +	struct buffer_head *bh, *head;
> >>>>  	struct inode *inode = page->mapping->host;
> >>>>
> >>>>  	handle = ext4_journal_current_handle();
> >>>> @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> >>>>  	else
> >>>>  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> >>>>
> >>>> +	 if (ret)
> >>>> +		 return ret;
> >>>> +	/*
> >>>> +	 * When called via shrink_page_list and if we don't have any unmapped
> >>>> +	 * buffer_head we still could have written some new content in an
> >>>> +	 * already mapped buffer. That means we need to extent i_disksize here
> >>>> +	 */
> >>>
> >>> In this case(when extend the file without need block allocation),
> >>> wouldn't make sense to update the i_disksize at write_end() time? So
> >>> that the window of i_size different from i_disksize could be much
> >>> smaller in this case.
> >>>
> >>>
> >>> Something like below? (untested)
> >>
> >> In this case you will have to start a transaction in write_begin . With
> >> the below code transaction is started inside page_lock. Also I don't
> >> think we need needed_blocks credit just 1 should be enough because we
> >> are not doing any block allocation here. We just need to update the
> >> inode block.
> >>
> >>
> >
> > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > index 33f940b..bc925c5 100644
> > --- a/fs/ext4/inode.c
> > +++ b/fs/ext4/inode.c
> > @@ -1770,6 +1770,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > 	struct page *page;
> > 	pgoff_t index;
> > 	unsigned from, to;
> > +	handle_t *handle;
> > 	struct inode *inode = mapping->host;
> >
> > 	index = pos >> PAGE_CACHE_SHIFT;
> > @@ -1777,6 +1778,17 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > 	to = from + len;
> >
> > retry:
> > +	/*
> > +	 * If we are writing towards the end of an already mapped
> > +	 * buffer_head, we don't do any block allocation. But we
> > +	 * need to update i_disksize.
> > +	 */
> > +	handle = ext4_journal_start(inode, 1);
> > +	if (IS_ERR(handle)) {
> > +		ret = PTR_ERR(handle);
> > +		goto out;
> > +	}
> > +
> > 	page = __grab_cache_page(mapping, index);
> > 	if (!page)
> > 		return -ENOMEM;
> > @@ -1786,15 +1798,63 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping,
> > 							ext4_da_get_block_prep);
> > 	if (ret < 0) {
> > 		unlock_page(page);
> > +		ext4_journal_stop(handle);
> > 		page_cache_release(page);
> > 	}
> >
> > 	if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries))
> > 		goto retry;
> >
> > +out:
> > 	return ret;
> > }
> >
> > +static int ext4_da_write_end(struct file *file,
> > +				struct address_space *mapping,
> > +				loff_t pos, unsigned len, unsigned copied,
> > +				struct page *page, void *fsdata)
> > +{
> > +	loff_t new_i_size;
> > +	unsigned from, to;
> > +	int ret = 0, ret2;
> > +	struct inode *inode = mapping->host;
> > +	handle_t *handle = ext4_journal_current_handle();
> > +
> > +	from = pos & (PAGE_CACHE_SIZE - 1);
> > +	to = from + len;
> > +
> > +	/*
> > +	 * generic_write_end() will run mark_inode_dirty() if i_size
> > +	 * changes.  So let's piggyback the i_disksize mark_inode_dirty
> > +	 * into that.
> > +	 */
> > +
> > +	new_i_size = pos + copied;
> > +	if (new_i_size > EXT4_I(inode)->i_disksize)
> > +		if (!walk_page_buffers(NULL, page_buffers(page),
> > +					0, len, NULL,
> > +					ext4_bh_unmapped_or_delay)) {
> > +			/*
> > +			 * Updating i_disksize when extending file without
> > +			 * need block allocation
> > +			 */
> > +			if (ext4_should_order_data(inode))
> > +				ret = ext4_jbd2_file_inode(handle, inode);
> > +
> > +			EXT4_I(inode)->i_disksize = new_i_size;
> > +		}
> > +	ret2 = generic_write_end(file, mapping, pos,
> > +			len, copied, page, fsdata);
> > +	copied = ret2;
> > +	if (ret2 < 0)
> > +		ret = ret2;
> > +	ret2 = ext4_journal_stop(handle);
> > +	if (!ret)
> > +		ret = ret2;
> > +
> > +	return ret ? ret : copied;
> > +}
> > +
> > static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> > {
> > 	/*
> > @@ -2250,7 +2310,7 @@ static int ext4_journalled_set_page_dirty(struct page *page)
> > 	.writepages	= ext4_da_writepages,
> > 	.sync_page	= block_sync_page,
> > 	.write_begin	= ext4_da_write_begin,
> > -	.write_end	= generic_write_end,
> > +	.write_end	= ext4_da_write_end,
> > 	.bmap		= ext4_bmap,
> > 	.invalidatepage	= ext4_da_invalidatepage,
> > 	.releasepage	= ext4_releasepage,
> >
> Yes, with this patch applied on top of latest patch queue I no longer
> get truncated files, after running a short test. Tomorrow I will do some
> more thorough testing and use the patch you have send to me in a separate
> mail. The above patch did not apply but it was easy to apply by hand.


Thanks for quick response and test. I have updated the patch queue with
above patch merged. Please let me know if you still see apply issue and
file size update issue with current patch queue.

Regards,
Mingming

Re: Performance of ext4

[Holger Kiehl]

On Tue, 24 Jun 2008, Mingming wrote:

>
> On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
>> Yes, with this patch applied on top of latest patch queue I no longer
>> get truncated files, after running a short test. Tomorrow I will do some
>> more thorough testing and use the patch you have send to me in a separate
>> mail. The above patch did not apply but it was easy to apply by hand.
>
>
> Thanks for quick response and test. I have updated the patch queue with
> above patch merged. Please let me know if you still see apply issue and
> file size update issue with current patch queue.
>
Thanks, it applies without any problems. However I still hit an oops. What
I find strange is that I got the oops just as the benchmark is done and
all process where shutting down. The same behaviour I reported here:
http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
and HW Raid). Anyway, here the dmesg output:

kjournald2 starting.  Commit interval 15 seconds
EXT4 FS on md7, internal journal
EXT4-fs: mounted filesystem with ordered data mode.
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
JBD: barrier-based sync failed on md7 - disabling barriers
------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1667!
invalid opcode: 0000 [1] SMP 
CPU 3 
Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy k8temp ohci_hcd i2c_amd756 i2c_core button usbcore
Pid: 2661, comm: kjournald2 Not tainted 2.6.26-rc6 #1
RIP: 0010:[<ffffffff802dd314>]  [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
RSP: 0018:ffff81027d737c10  EFLAGS: 00010246
RAX: 010000000001002d RBX: ffffe200017e9218 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff81027d737e00 RDI: ffffe200017e9218
RBP: ffff81021dad66e8 R08: ffff81027ff05440 R09: ffff81027d737b50
R10: ffff81020fe54480 R11: ffff81020fe54400 R12: ffff81027d737e00
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8101ff0b8120
FS:  00007f53ed10b6f0(0000) GS:ffff81027ff13000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000000081a000 CR3: 0000000166f03000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kjournald2 (pid: 2661, threadinfo ffff81027d736000, task ffff81027ed93210)
Stack:  ffff81021dad67f8 ffff81027d737e00 0000000000000000 ffffffff80261816
  ffffe200017e9218 ffffffff80261f8c ffff81021dad67f8 ffffffff8026180c
  ffff81021dad67f8 0000000280224f0c 0000000000000004 000000007d737fd8
Call Trace:
  [<ffffffff80261816>] ? __writepage+0xa/0x27
  [<ffffffff80261f8c>] ? write_cache_pages+0x174/0x2be
  [<ffffffff8026180c>] ? __writepage+0x0/0x27
  [<ffffffff802ff1ed>] ? jbd2_journal_commit_transaction+0x343/0xe6a
  [<ffffffff802411a8>] ? hrtimer_start+0x100/0x122
  [<ffffffff802357ba>] ? try_to_del_timer_sync+0x52/0x5b
  [<ffffffff80302972>] ? kjournald2+0xdf/0x202
  [<ffffffff8023e74a>] ? autoremove_wake_function+0x0/0x2e
  [<ffffffff80302893>] ? kjournald2+0x0/0x202
  [<ffffffff8023e43b>] ? kthread+0x47/0x73
  [<ffffffff8020bf78>] ? child_rip+0xa/0x12
  [<ffffffff8023e3f4>] ? kthread+0x0/0x73
  [<ffffffff8020bf6e>] ? child_rip+0x0/0x12


Code: 55 53 48 8b 47 18 48 89 fb 48 8b 28 65 48 8b 04 25 00 00 00 00 48 83 b8 d0 04 00 00 00 75 5f 48 8b 07 48 8b 55 68 f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c 
RIP  [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
  RSP <ffff81027d737c10>
---[ end trace efaf3891d582feb1 ]---
SysRq : Show Blocked State
   task                        PC stack   pid father
pdflush       D 00000000ffffffff     0   242      2
  ffff81007f319bb8 0000000000000046 ffffffff8031682e 000000000000000e
  ffffffff80600cc0 000000000000000e ffff81007f319bf0 ffff81007fbbbd30
  ffff8101fff33210 ffff81007fbbbf68 00000002802aa2a6 ffff81007fbbbf68
Call Trace:
  [<ffffffff8031682e>] submit_bio+0xc4/0xcb
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fde7f>] jbd2_journal_stop+0x191/0x19d
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dd7e8>] ext4_da_writepages+0x75/0x173
  [<ffffffff80262112>] do_writepages+0x20/0x2d
  [<ffffffff8029f6da>] __writeback_single_inode+0x16f/0x35e
  [<ffffffff8029fcf7>] sync_sb_inodes+0x283/0x3b8
  [<ffffffff802a0080>] writeback_inodes+0x89/0xde
  [<ffffffff80262252>] wb_kupdate+0x9f/0x116
  [<ffffffff80262ca0>] pdflush+0x122/0x1c9
  [<ffffffff802621b3>] wb_kupdate+0x0/0x116
  [<ffffffff80262b7e>] pdflush+0x0/0x1c9
  [<ffffffff8023e43b>] kthread+0x47/0x73
  [<ffffffff8022ae6a>] schedule_tail+0x27/0x5c
  [<ffffffff8020bf78>] child_rip+0xa/0x12
  [<ffffffff8023e3f4>] kthread+0x0/0x73
  [<ffffffff8020bf6e>] child_rip+0x0/0x12

init_afd      D 00000000ffffffff     0  7974      1
  ffff8101050d3b30 0000000000000082 ffff810055f08230 0000000000080768
  ffffffff80600cc0 ffff8101ff8509c0 ffffffff802a38d5 ffff8101050aac80
  ffff8101fff342c0 ffff8101050aaeb8 000000038023e66b ffff8101050aaeb8
Call Trace:
  [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80224a97>] __wake_up+0x38/0x4f
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
  [<ffffffff80240fcb>] lock_hrtimer_base+0x1b/0x3c
  [<ffffffff80241089>] hrtimer_try_to_cancel+0x61/0x6a
  [<ffffffff8024109e>] hrtimer_cancel+0xc/0x16
  [<ffffffff8046687e>] do_nanosleep+0x6c/0xa6
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

system_log    D 00000000ffffffff     0  7975   7974
  ffff81002d533ad0 0000000000000086 ffffffff8028fce8 ffff81007ed55e90
  ffffffff80600cc0 0000000000000100 ffff81027d7cae40 ffff81007ed55e90
  ffff8101fff342c0 ffff81007ed560c8 000000032d533f50 ffff81007ed560c8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8026f499>] mmap_region+0x41a/0x505
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

receive_log   D 00000000ffffffff     0  7977   7974
  ffff81002d4ffad0 0000000000000082 ffffffff8028fce8 ffff81007ed55900
  ffffffff80600cc0 0000000000000100 ffff81007e86ed80 ffff81007ed55900
  ffffffff8054d350 ffff81007ed55b38 000000002d4fff50 ffff81007ed55b38
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff     0  7980   7974
  ffff81007ecabce0 0000000000000082 ffffffff802de7cd ffff810000000001
  ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007ed537a0
  ffff8101fff33210 ffff81007ed539d8 00000002802d9b7a ffff81007ed539d8
Call Trace:
  [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
  [<ffffffff8027ce42>] get_partial_node+0x15/0x78
  [<ffffffff80294b15>] __d_lookup+0xbd/0x107
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
  [<ffffffff8028b364>] may_delete+0x5e/0x135
  [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
  [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8020b26e>] tracesys+0x71/0xda
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

amg           D ffff81007e8b3cf8     0  7986   7974
  ffff81002d50bb60 0000000000000086 0000000000000000 0000000000000100
  ffffffff80600cc0 0000000000000000 0000000000000000 ffff81007ed50000
  ffff8101050acde0 ffff81007ed50238 0000000100000000 ffff81007ed50238
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     D ffff81007e8b1690     0  7988   7986
  ffff81002d625d70 0000000000000082 ffff81002d625d38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81007ed542c0
  ffff81007ed574d0 ffff81007ed544f8 000000028028c917 ffff81007ed544f8
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat      D 00000000ffffffff     0  7989   7974
  ffff81002d461b30 0000000000000082 ffff81002d461da8 ffff81002d461da8
  ffffffff80600cc0 ffff81002d461da8 ffffffff802a38d5 ffff81007ed57a60
  ffffffff8054d350 ffff81007ed57c98 000000008023e66b ffff81007ed57c98
Call Trace:
  [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80262dda>] pdflush_operation+0x93/0x9d
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
  [<ffffffff802411a8>] hrtimer_start+0x100/0x122
  [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

init_afd      D ffff81007e8b3cf8     0  7997      1
  ffff81027ee41b30 0000000000000086 ffff81027ee41dd0 ffff81027ee41da8
  ffffffff80600cc0 ffff81027ee41db8 0000000000000010 ffff81027ecca160
  ffff8101bbec1640 ffff81027ecca398 0000000000000000 ffff81027ecca398
Call Trace:
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

transfer_log  D 00000000ffffffff     0  8001   7997
  ffff81002d681ad0 0000000000000082 ffffffff8028fce8 ffff81007fb8f4d0
  ffffffff80600cc0 0000000000000100 ffff8101b1362540 ffff81007fb8f4d0
  ffff8101fff33210 ffff81007fb8f708 000000022d681f50 ffff81007fb8f708
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff     0  8003   7997
  ffff81002d49dce0 0000000000000086 ffffffff802de7cd ffff810000000001
  ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007fb8ef40
  ffff8101fff32160 ffff81007fb8f178 00000001802d9b7a ffff81007fb8f178
Call Trace:
  [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
  [<ffffffff8027ce42>] get_partial_node+0x15/0x78
  [<ffffffff80294b15>] __d_lookup+0xbd/0x107
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
  [<ffffffff8028b364>] may_delete+0x5e/0x135
  [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
  [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8020b26e>] tracesys+0x71/0xda
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log     D 00000000ffffffff     0  8004   7997
  ffff81002d419ad0 0000000000000086 ffffffff8028fce8 ffff81007fb8de90
  ffffffff80600cc0 0000000000000100 ffff81027f76aa80 ffff81007fb8de90
  ffff8101fff32160 ffff81007fb8e0c8 000000012d419f50 ffff81007fb8e0c8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log    D ffff81007e8b3cf8     0  8005   7997
  ffff81002d4fdad0 0000000000000086 ffffffff8028fce8 ffff81007fbbc2c0
  ffffffff80600cc0 0000000000000100 ffff81017ff649c0 ffff81007fbbc2c0
  ffff81001ce46420 ffff81007fbbc4f8 000000012d4fdf50 ffff81007fbbc4f8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

production_lo D ffff81007e8b3cf8     0  8007   7997
  ffff81002d615ad0 0000000000000082 ffffffff8028fce8 ffff81007d939bd0
  ffffffff80600cc0 0000000000000100 ffff81027f76a3c0 ffff81007d939bd0
  ffff81007ed50590 ffff81007d939e08 000000012d615f50 ffff81007d939e08
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8026f499>] mmap_region+0x41a/0x505
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ D 00000000ffffffff     0  8008   7997
  ffff81002d53fad0 0000000000000086 ffffffff8028fce8 ffff81007d93ac80
  ffffffff80600cc0 0000000000000100 ffff81007f300600 ffff81007d93ac80
  ffff8101fff33210 ffff81007d93aeb8 000000022d53ff50 ffff81007d93aeb8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     D ffff81007e8b1690     0  8011   8009
  ffff8101e1359d70 0000000000000082 ffff8101e1359d38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec10b0
  ffff81007fb8de90 ffff8101bbec12e8 000000018028c917 ffff8101bbec12e8
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat      D ffff81007e8b3cf8     0  8012   7997
  ffff81002d69bb30 0000000000000082 ffff81002d69bda8 ffff81002d69bda8
  ffffffff80600cc0 ffff81002d69bda8 ffffffff802a38d5 ffff81007d93c2c0
  ffff81027eccd900 ffff81007d93c4f8 000000008023e66b ffff81007d93c4f8
Call Trace:
  [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80224a97>] __wake_up+0x38/0x4f
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
  [<ffffffff802411a8>] hrtimer_start+0x100/0x122
  [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

init_afd      D 00000000ffffffff     0  8020      1
  ffff8101bbdd5b30 0000000000000086 ffff8101bbdd5dd0 ffff8101bbdd5da8
  ffffffff80600cc0 ffff8101bbdd5db8 0000000000000010 ffff8101bbec1640
  ffffffff8054d350 ffff8101bbec1878 0000000000000000 ffff8101bbec1878
Call Trace:
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

receive_log   D 00000000ffffffff     0  8023   8020
  ffff81027ee1bad0 0000000000000086 ffffffff8028fce8 ffff81027eccc2c0
  ffffffff80600cc0 0000000000000100 ffff81007f300000 ffff81027eccc2c0
  ffff8101fff33210 ffff81027eccc4f8 000000027ee1bf50 ffff81027eccc4f8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

transfer_log  D 00000000ffffffff     0  8024   8020
  ffff81027f799ad0 0000000000000086 ffffffff8028fce8 ffff81027eccc850
  ffffffff80600cc0 0000000000000100 ffff81027d7443c0 ffff81027eccc850
  ffffffff8054d350 ffff81027eccca88 000000007f799f50 ffff81027eccca88
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

archive_watch D 00000000ffffffff     0  8026   8020
  ffff81027f7ddce0 0000000000000082 ffffffff802de7cd ffff810200000001
  ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81027ecc90b0
  ffff8101fff33210 ffff81027ecc92e8 00000002802d9b7a ffff81027ecc92e8
Call Trace:
  [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
  [<ffffffff8027ce42>] get_partial_node+0x15/0x78
  [<ffffffff80294b15>] __d_lookup+0xbd/0x107
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
  [<ffffffff8028b364>] may_delete+0x5e/0x135
  [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
  [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8020b26e>] tracesys+0x71/0xda
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

input_log     D ffff81007e8b3cf8     0  8027   8020
  ffff81027eed1ad0 0000000000000082 ffffffff8028fce8 ffff81027ecca6f0
  ffffffff80600cc0 0000000000000100 ffff81027d744e40 ffff81027ecca6f0
  ffff81007fb8de90 ffff81027ecca928 000000017eed1f50 ffff81027ecca928
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

output_log    D ffff81007e8b3cf8     0  8028   8020
  ffff81027d64dad0 0000000000000086 ffffffff8028fce8 ffff81027eccac80
  ffffffff80600cc0 0000000000000100 ffff81010508c840 ffff81027eccac80
  ffff81007ed574d0 ffff81027eccaeb8 000000027d64df50 ffff81027eccaeb8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

distribution_ D 00000000ffffffff     0  8031   8020
  ffff81027ee1dad0 0000000000000086 ffffffff8028fce8 ffff81027ecc8590
  ffffffff80600cc0 0000000000000100 ffff810105128a80 ffff81027ecc8590
  ffff8101fff33210 ffff81027ecc87c8 000000027ee1df50 ffff81027ecc87c8
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
  [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80295a76>] touch_atime+0x12/0xfa
  [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

dir_check     D 00000000ffffffff     0  8034   8032
  ffff81027ec01d70 0000000000000082 ffff81027ec01d38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81027eccb210
  ffff8101fff33210 ffff81027eccb448 000000028028c917 ffff81027eccb448
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

afd_stat      D ffff81007e8b3cf8     0  8035   8020
  ffff81027edebb30 0000000000000082 ffff81027edebda8 ffff81027edebda8
  ffffffff80600cc0 ffff81027edebda8 ffffffff802a38d5 ffff81027eccd900
  ffff81007ed57a60 ffff81027eccdb38 000000008023e66b ffff81027eccdb38
Call Trace:
  [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80224a97>] __wake_up+0x38/0x4f
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
  [<ffffffff802411a8>] hrtimer_start+0x100/0x122
  [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

fd            D ffff81007e8b3cf8     0  8037   8020
  ffff81027ededb30 0000000000000086 ffff81027ededdd0 ffff81027ededda8
  ffffffff80600cc0 ffff81027ededdb8 00000000000007e0 ffff81027eccde90
  ffff81027ed937a0 ffff81027ecce0c8 0000000200000000 ffff81027ecce0c8
Call Trace:
  [<ffffffff80290dfd>] __pollwait+0x0/0xe3
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff8022f67f>] release_task+0x2cb/0x2dd
  [<ffffffff80230175>] do_wait+0xae4/0xb88
  [<ffffffff80286e19>] vfs_stat_fd+0x1b/0x4a
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

afd           D 00000000ffffffff     0 25970   2891
  ffff81017fae7ad0 0000000000000086 ffffffff8028fce8 0000000000000246
  ffffffff80600cc0 0000000000000100 ffff81020fe5dc00 ffff8101050ab210
  ffff8101fff342c0 ffff8101050ab448 000000037fae7f50 ffff8101050ab448
Call Trace:
  [<ffffffff8028fce8>] poll_freewait+0x36/0x87
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80268f85>] __do_fault+0x321/0x369
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80237d9a>] group_send_sig_info+0x62/0x6f
  [<ffffffff80237dd8>] kill_pid_info+0x31/0x3b
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b3cf8     0 26537   8037
  ffff8101ff7b1b70 0000000000000086 ffffffff802411a8 000000000000148d
  ffffffff80600cc0 0000000000000286 00000000ffffffff ffff8101bbec26f0
  ffff81027eccb210 ffff8101bbec2928 00000002ffffffff ffff8101bbec2928
Call Trace:
  [<ffffffff802411a8>] hrtimer_start+0x100/0x122
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80466148>] thread_return+0x63/0xbb
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
  [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
  [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b1690     0 26543   8037
  ffff8101ddec3d70 0000000000000086 ffff8101ddec3d38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec3210
  ffff81007ed542c0 ffff8101bbec3448 000000028028c917 ffff8101bbec3448
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b3cf8     0 26559   8037
  ffff8101b11ddb60 0000000000000086 ffff8101bfc0b4d0 0000000000000202
  ffffffff80600cc0 ffff8101bfc0b4d0 00007f1ea612f000 ffff8101bbec74d0
  ffff81007ed574d0 ffff8101bbec7708 0000000200000000 ffff8101bbec7708
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80467779>] error_exit+0x0/0x51
  [<ffffffff8025bae5>] find_lock_page+0x29/0x87
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d535>] unmap_region+0x10f/0x125
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 26569   8013
  ffff8101f40c1b60 0000000000000082 ffff81013b896b00 0000000000000206
  ffffffff80600cc0 ffff81013b896b00 00007f31838c7000 ffff8101bbec6f40
  ffff8101fff32160 ffff8101bbec7178 0000000100000000 ffff8101bbec7178
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80467779>] error_exit+0x0/0x51
  [<ffffffff8025bae5>] find_lock_page+0x29/0x87
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d535>] unmap_region+0x10f/0x125
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b3cf8     0 26583   8013
  ffff8101bbf9fb60 0000000000000086 ffff81010c703160 0000000000000202
  ffffffff80600cc0 ffff81010c703160 00007ffac6bd2000 ffff8101bbec6420
  ffff81007d93fa60 ffff8101bbec6658 0000000100000000 ffff8101bbec6658
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80467779>] error_exit+0x0/0x51
  [<ffffffff8025bae5>] find_lock_page+0x29/0x87
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d535>] unmap_region+0x10f/0x125
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff810208bf59f8     0 26589  26567
  ffff810208bf59d8 0000000000000082 ffff810163233090 ffffffff802e9b05
  ffffffff80600cc0 ffffffff00058011 ffff810208bf5af8 ffff81027ecce420
  ffff81001ce40000 ffff81027ecce658 0000000208bf5b58 ffff81027ecce658
Call Trace:
  [<ffffffff802e9b05>] ext4_ext_find_extent+0x67/0x26f
  [<ffffffff80224a97>] __wake_up+0x38/0x4f
  [<ffffffff803023b0>] jbd2_log_wait_commit+0xb6/0x108
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80300c3c>] jbd2_log_do_checkpoint+0x12d/0x3bd
  [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
  [<ffffffff802a3c4d>] __find_get_block+0x14c/0x15c
  [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
  [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
  [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
  [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
  [<ffffffff802e1b3a>] ext4_find_entry+0x293/0x592
  [<ffffffff80300f53>] __jbd2_log_wait_for_space+0x87/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8029535a>] d_alloc+0x1f/0x189
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e24bd>] ext4_create+0x4c/0x105
  [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
  [<ffffffff8028b805>] vfs_create+0x7d/0xed
  [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
  [<ffffffff8028290c>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff81007e8b3cf8     0 26595  26584
  ffff8101b1307ad0 0000000000000086 ffffffff8042530e ffff81014256f5d0
  ffffffff80600cc0 ffff81017fa8fa00 ffffffff80242fdc ffff8101ff6d8b20
  ffff81001ce46420 ffff8101ff6d8d58 000000018024153d ffff8101ff6d8d58
Call Trace:
  [<ffffffff8042530e>] nf_iterate+0x41/0x7d
  [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
  [<ffffffff8039d0b6>] loopback_xmit+0x80/0x86
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff81007e8b3cf8     0 26602  26590
  ffff81025dcf1c60 0000000000000082 ffffffff802e6fb3 0000000000000020
  ffffffff80600cc0 ffff81014256f090 00000000000300aa ffff81027ed942c0
  ffff81007d93e420 ffff81027ed944f8 000000017ae2b000 ffff81027ed944f8
Call Trace:
  [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
  [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e24bd>] ext4_create+0x4c/0x105
  [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
  [<ffffffff8028b805>] vfs_create+0x7d/0xed
  [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
  [<ffffffff8028290c>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 26605   8013
  ffff81001cdc9b60 0000000000000086 ffff8101bfc0b4d0 0000000000000216
  ffffffff80600cc0 ffff8101bfc0b4d0 00007f628a734000 ffff81001ce43d30
  ffff8101fff33210 ffff81001ce43f68 0000000200000000 ffff81001ce43f68
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80467779>] error_exit+0x0/0x51
  [<ffffffff8025bae5>] find_lock_page+0x29/0x87
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d535>] unmap_region+0x10f/0x125
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b1690     0 26608   8013
  ffff81003eb4fd70 0000000000000082 ffff81003eb4fd38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce43210
  ffff81001ce410b0 ffff81001ce43448 000000018028c917 ffff81001ce43448
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b1690     0 26609   8013
  ffff81003eb63d70 0000000000000082 ffff81003eb63d38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce442c0
  ffff81017faea160 ffff81001ce444f8 000000008028c917 ffff81001ce444f8
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b1690     0 26610   8013
  ffff81003e98dd70 0000000000000082 ffff81003e98dd38 ffff81007edfcb00
  ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce42c80
  ffff81017fae8590 ffff81001ce42eb8 000000028028c917 ffff81001ce42eb8
Call Trace:
  [<ffffffff80293dd5>] dput+0x21/0x10a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028ac53>] lock_rename+0x35/0xc5
  [<ffffffff8028d545>] do_rename+0x98/0x1b2
  [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
  [<ffffffff8027ce42>] get_partial_node+0x15/0x78
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 26611   8013
  ffff81001ced3b60 0000000000000082 ffff81002d7fa0d0 ffff81002d7fa0d0
  ffffffff80600cc0 00000000486202ae 0000000210e7c865 ffff81001ce41640
  ffffffff8054d350 ffff81001ce41878 0000000000000003 ffff81001ce41878
Call Trace:
  [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff80282afc>] __dentry_open+0x141/0x226
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D 00000000ffffffff     0 26613   8013
  ffff8101e1321b30 0000000000000082 0000000000000296 00000000662df001
  ffffffff80600cc0 ffff8101d40c9e90 ffffffff80242fdc ffff8101ff6dc850
  ffff8101fff33210 ffff8101ff6dca88 000000028024153d ffff8101ff6dca88
Call Trace:
  [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffffa0070d34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
  [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467779>] error_exit+0x0/0x51

vsftpd        D ffff81007e8b3cf8     0 26614  26606
  ffff8100588b1c90 0000000000000086 ffffffff802411a8 000000000000148d
  ffffffff80600cc0 0000000000000286 00000000ffffffff ffff81001ce40000
  ffff8101ff6dd900 ffff81001ce40238 00000002ffffffff ffff81001ce40238
Call Trace:
  [<ffffffff802411a8>] hrtimer_start+0x100/0x122
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80466148>] thread_return+0x63/0xbb
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80296e64>] inode_setattr+0x103/0x10a
  [<ffffffff802dc9b0>] ext4_setattr+0x1eb/0x244
  [<ffffffff80296fbb>] notify_change+0x150/0x26d
  [<ffffffff80283568>] do_truncate+0x5e/0x79
  [<ffffffff80256c42>] audit_syscall_entry+0x11f/0x152
  [<ffffffff80283664>] sys_ftruncate+0xe1/0xfd
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

bzip2         D 00000000ffffffff     0 26616  26615
  ffff8102043d9ad0 0000000000000082 ffffffff80278233 ffff810256ebc660
  ffffffff80600cc0 00007f5eba717000 0000000000000002 ffff81027ed91640
  ffff8101fff342c0 ffff81027ed91878 000000038026a789 ffff81027ed91878
Call Trace:
  [<ffffffff80278233>] alloc_page_vma+0x154/0x221
  [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
  [<ffffffff80266f91>] zone_statistics+0x3c/0x90
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80268f85>] __do_fault+0x321/0x369
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
  [<ffffffff80466148>] thread_return+0x63/0xbb
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

sf_ftp        D ffff81007e8b3cf8     0 26621   8037
  ffff8101b1103b60 0000000000000082 ffff81013b896b00 0000000000000206
  ffffffff80600cc0 ffff81013b896b00 00007f4f1e28e000 ffff8101ff6df4d0
  ffff8101bbec6f40 ffff8101ff6df708 0000000100000000 ffff8101ff6df708
Call Trace:
  [<ffffffff80224f1d>] default_wake_function+0x0/0xe
  [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80467779>] error_exit+0x0/0x51
  [<ffffffff8025bae5>] find_lock_page+0x29/0x87
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
  [<ffffffff8026d535>] unmap_region+0x10f/0x125
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D ffff81007e8b3cf8     0 26625  26617
  ffff810039cf9b70 0000000000000082 ffff810270718a80 ffff81022646e2a0
  ffffffff80600cc0 ffff810039cf9d44 0000000000000000 ffff81001ce469b0
  ffff81001ce41640 ffff81001ce46be8 00000000802a3c7b ffff81001ce46be8
Call Trace:
  [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80294b15>] __d_lookup+0xbd/0x107
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
  [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
  [<ffffffff802614c3>] __alloc_pages_internal+0xd2/0x40c
  [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0 26632  26626
  ffff8101dddc3c60 0000000000000082 ffffffff802e6fb3 0000000000000020
  ffffffff80600cc0 ffff8101d40c8610 0000000000031014 ffff8101ff6dd900
  ffff8101fff33210 ffff8101ff6ddb38 000000021d4e6000 ffff8101ff6ddb38
Call Trace:
  [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
  [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802e24bd>] ext4_create+0x4c/0x105
  [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
  [<ffffffff8028b805>] vfs_create+0x7d/0xed
  [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
  [<ffffffff8028290c>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

vsftpd        D 00000000ffffffff     0 26635  26633
  ffff8100066c9dd0 0000000000000086 ffffffff804082ac ffff8100066c9d34
  ffffffff80600cc0 0000000e9aa3566f ffff81007edfcbc8 ffff81007d93e420
  ffff8101fff32160 ffff81007d93e658 0000000100000000 ffff81007d93e658
Call Trace:
  [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
  [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff8028e2cb>] do_filp_open+0x10c/0x832
  [<ffffffff80282837>] get_unused_fd_flags+0x7f/0x10e
  [<ffffffff8028290c>] do_sys_open+0x46/0xc3
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

bzip2         D 00000000ffffffff     0 26638  26637
  ffff8101427cdad0 0000000000000082 ffffffff80278233 ffff8101f6fa70c0
  ffffffff80600cc0 00007f0c65f4e000 0000000000000002 ffff81017faec850
  ffffffff8054d350 ffff81017faeca88 000000008026a789 ffff81017faeca88
Call Trace:
  [<ffffffff80278233>] alloc_page_vma+0x154/0x221
  [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
  [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
  [<ffffffff80466561>] mutex_lock+0xa/0xb
  [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
  [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
  [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
  [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
  [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
  [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
  [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
  [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
  [<ffffffff80268f85>] __do_fault+0x321/0x369
  [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
  [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
  [<ffffffff80283eb3>] do_sync_write+0xce/0x113
  [<ffffffff8021ed48>] do_page_fault+0x471/0x825
  [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
  [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
  [<ffffffff80466148>] thread_return+0x63/0xbb
  [<ffffffff802846d8>] vfs_write+0xad/0x136
  [<ffffffff8028481d>] sys_write+0x45/0x6e
  [<ffffffff8020b2d2>] tracesys+0xd5/0xda

I used ext4-patch-queue-b5db22ef52ed53d8e3fa978a5a29e1609c9333aa.tar.gz and
a clean linux-2.6.26-rc6 with no other patches.

Holger

Re: Performance of ext4

[Mingming]

On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> On Tue, 24 Jun 2008, Mingming wrote:
> 
> >
> > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> >> Yes, with this patch applied on top of latest patch queue I no longer
> >> get truncated files, after running a short test. Tomorrow I will do some
> >> more thorough testing and use the patch you have send to me in a separate
> >> mail. The above patch did not apply but it was easy to apply by hand.
> >
> >
> > Thanks for quick response and test. I have updated the patch queue with
> > above patch merged. Please let me know if you still see apply issue and
> > file size update issue with current patch queue.
> >
> Thanks, it applies without any problems. However I still hit an oops. What
> I find strange is that I got the oops just as the benchmark is done and
> all process where shutting down. The same behaviour I reported here:
> http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> and HW Raid). Anyway, here the dmesg output:
> 
> kjournald2 starting.  Commit interval 15 seconds
> EXT4 FS on md7, internal journal
> EXT4-fs: mounted filesystem with ordered data mode.
> EXT4-fs: file extents enabled
> EXT4-fs: mballoc enabled
> JBD: barrier-based sync failed on md7 - disabling barriers
> ------------[ cut here ]------------
> kernel BUG at fs/ext4/inode.c:1667!

Did not get a chance to look more closely today, but it's point to this
code in ext4_da_writepage()

 page_bufs = page_buffers(page);

and appearently it's BUG_ON at 
 BUG_ON(!PagePrivate(page)); in page_buffers().


Mingming

> invalid opcode: 0000 [1] SMP 
> CPU 3 
> Modules linked in: w83627hf lm85 hwmon_vid bonding nf_conntrack_ftp ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy k8temp ohci_hcd i2c_amd756 i2c_core button usbcore
> Pid: 2661, comm: kjournald2 Not tainted 2.6.26-rc6 #1
> RIP: 0010:[<ffffffff802dd314>]  [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
> RSP: 0018:ffff81027d737c10  EFLAGS: 00010246
> RAX: 010000000001002d RBX: ffffe200017e9218 RCX: 00000000ffffffe8
> RDX: 0000000000004278 RSI: ffff81027d737e00 RDI: ffffe200017e9218
> RBP: ffff81021dad66e8 R08: ffff81027ff05440 R09: ffff81027d737b50
> R10: ffff81020fe54480 R11: ffff81020fe54400 R12: ffff81027d737e00
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff8101ff0b8120
> FS:  00007f53ed10b6f0(0000) GS:ffff81027ff13000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 000000000081a000 CR3: 0000000166f03000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process kjournald2 (pid: 2661, threadinfo ffff81027d736000, task ffff81027ed93210)
> Stack:  ffff81021dad67f8 ffff81027d737e00 0000000000000000 ffffffff80261816
>   ffffe200017e9218 ffffffff80261f8c ffff81021dad67f8 ffffffff8026180c
>   ffff81021dad67f8 0000000280224f0c 0000000000000004 000000007d737fd8
> Call Trace:
>   [<ffffffff80261816>] ? __writepage+0xa/0x27
>   [<ffffffff80261f8c>] ? write_cache_pages+0x174/0x2be
>   [<ffffffff8026180c>] ? __writepage+0x0/0x27
>   [<ffffffff802ff1ed>] ? jbd2_journal_commit_transaction+0x343/0xe6a
>   [<ffffffff802411a8>] ? hrtimer_start+0x100/0x122
>   [<ffffffff802357ba>] ? try_to_del_timer_sync+0x52/0x5b
>   [<ffffffff80302972>] ? kjournald2+0xdf/0x202
>   [<ffffffff8023e74a>] ? autoremove_wake_function+0x0/0x2e
>   [<ffffffff80302893>] ? kjournald2+0x0/0x202
>   [<ffffffff8023e43b>] ? kthread+0x47/0x73
>   [<ffffffff8020bf78>] ? child_rip+0xa/0x12
>   [<ffffffff8023e3f4>] ? kthread+0x0/0x73
>   [<ffffffff8020bf6e>] ? child_rip+0x0/0x12
> 
> 
> Code: 55 53 48 8b 47 18 48 89 fb 48 8b 28 65 48 8b 04 25 00 00 00 00 48 83 b8 d0 04 00 00 00 75 5f 48 8b 07 48 8b 55 68 f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c 
> RIP  [<ffffffff802dd314>] ext4_da_writepage+0x30/0xf4
>   RSP <ffff81027d737c10>
> ---[ end trace efaf3891d582feb1 ]---
> SysRq : Show Blocked State
>    task                        PC stack   pid father
> pdflush       D 00000000ffffffff     0   242      2
>   ffff81007f319bb8 0000000000000046 ffffffff8031682e 000000000000000e
>   ffffffff80600cc0 000000000000000e ffff81007f319bf0 ffff81007fbbbd30
>   ffff8101fff33210 ffff81007fbbbf68 00000002802aa2a6 ffff81007fbbbf68
> Call Trace:
>   [<ffffffff8031682e>] submit_bio+0xc4/0xcb
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fde7f>] jbd2_journal_stop+0x191/0x19d
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dd7e8>] ext4_da_writepages+0x75/0x173
>   [<ffffffff80262112>] do_writepages+0x20/0x2d
>   [<ffffffff8029f6da>] __writeback_single_inode+0x16f/0x35e
>   [<ffffffff8029fcf7>] sync_sb_inodes+0x283/0x3b8
>   [<ffffffff802a0080>] writeback_inodes+0x89/0xde
>   [<ffffffff80262252>] wb_kupdate+0x9f/0x116
>   [<ffffffff80262ca0>] pdflush+0x122/0x1c9
>   [<ffffffff802621b3>] wb_kupdate+0x0/0x116
>   [<ffffffff80262b7e>] pdflush+0x0/0x1c9
>   [<ffffffff8023e43b>] kthread+0x47/0x73
>   [<ffffffff8022ae6a>] schedule_tail+0x27/0x5c
>   [<ffffffff8020bf78>] child_rip+0xa/0x12
>   [<ffffffff8023e3f4>] kthread+0x0/0x73
>   [<ffffffff8020bf6e>] child_rip+0x0/0x12
> 
> init_afd      D 00000000ffffffff     0  7974      1
>   ffff8101050d3b30 0000000000000082 ffff810055f08230 0000000000080768
>   ffffffff80600cc0 ffff8101ff8509c0 ffffffff802a38d5 ffff8101050aac80
>   ffff8101fff342c0 ffff8101050aaeb8 000000038023e66b ffff8101050aaeb8
> Call Trace:
>   [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
>   [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80224a97>] __wake_up+0x38/0x4f
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
>   [<ffffffff80240fcb>] lock_hrtimer_base+0x1b/0x3c
>   [<ffffffff80241089>] hrtimer_try_to_cancel+0x61/0x6a
>   [<ffffffff8024109e>] hrtimer_cancel+0xc/0x16
>   [<ffffffff8046687e>] do_nanosleep+0x6c/0xa6
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> system_log    D 00000000ffffffff     0  7975   7974
>   ffff81002d533ad0 0000000000000086 ffffffff8028fce8 ffff81007ed55e90
>   ffffffff80600cc0 0000000000000100 ffff81027d7cae40 ffff81007ed55e90
>   ffff8101fff342c0 ffff81007ed560c8 000000032d533f50 ffff81007ed560c8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8026f499>] mmap_region+0x41a/0x505
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> receive_log   D 00000000ffffffff     0  7977   7974
>   ffff81002d4ffad0 0000000000000082 ffffffff8028fce8 ffff81007ed55900
>   ffffffff80600cc0 0000000000000100 ffff81007e86ed80 ffff81007ed55900
>   ffffffff8054d350 ffff81007ed55b38 000000002d4fff50 ffff81007ed55b38
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> archive_watch D 00000000ffffffff     0  7980   7974
>   ffff81007ecabce0 0000000000000082 ffffffff802de7cd ffff810000000001
>   ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007ed537a0
>   ffff8101fff33210 ffff81007ed539d8 00000002802d9b7a ffff81007ed539d8
> Call Trace:
>   [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
>   [<ffffffff8027ce42>] get_partial_node+0x15/0x78
>   [<ffffffff80294b15>] __d_lookup+0xbd/0x107
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
>   [<ffffffff8028b364>] may_delete+0x5e/0x135
>   [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
>   [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8020b26e>] tracesys+0x71/0xda
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> amg           D ffff81007e8b3cf8     0  7986   7974
>   ffff81002d50bb60 0000000000000086 0000000000000000 0000000000000100
>   ffffffff80600cc0 0000000000000000 0000000000000000 ffff81007ed50000
>   ffff8101050acde0 ffff81007ed50238 0000000100000000 ffff81007ed50238
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> dir_check     D ffff81007e8b1690     0  7988   7986
>   ffff81002d625d70 0000000000000082 ffff81002d625d38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81007ed542c0
>   ffff81007ed574d0 ffff81007ed544f8 000000028028c917 ffff81007ed544f8
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> afd_stat      D 00000000ffffffff     0  7989   7974
>   ffff81002d461b30 0000000000000082 ffff81002d461da8 ffff81002d461da8
>   ffffffff80600cc0 ffff81002d461da8 ffffffff802a38d5 ffff81007ed57a60
>   ffffffff8054d350 ffff81007ed57c98 000000008023e66b ffff81007ed57c98
> Call Trace:
>   [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
>   [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80262dda>] pdflush_operation+0x93/0x9d
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
>   [<ffffffff802411a8>] hrtimer_start+0x100/0x122
>   [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> init_afd      D ffff81007e8b3cf8     0  7997      1
>   ffff81027ee41b30 0000000000000086 ffff81027ee41dd0 ffff81027ee41da8
>   ffffffff80600cc0 ffff81027ee41db8 0000000000000010 ffff81027ecca160
>   ffff8101bbec1640 ffff81027ecca398 0000000000000000 ffff81027ecca398
> Call Trace:
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> transfer_log  D 00000000ffffffff     0  8001   7997
>   ffff81002d681ad0 0000000000000082 ffffffff8028fce8 ffff81007fb8f4d0
>   ffffffff80600cc0 0000000000000100 ffff8101b1362540 ffff81007fb8f4d0
>   ffff8101fff33210 ffff81007fb8f708 000000022d681f50 ffff81007fb8f708
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> archive_watch D 00000000ffffffff     0  8003   7997
>   ffff81002d49dce0 0000000000000086 ffffffff802de7cd ffff810000000001
>   ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81007fb8ef40
>   ffff8101fff32160 ffff81007fb8f178 00000001802d9b7a ffff81007fb8f178
> Call Trace:
>   [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
>   [<ffffffff8027ce42>] get_partial_node+0x15/0x78
>   [<ffffffff80294b15>] __d_lookup+0xbd/0x107
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
>   [<ffffffff8028b364>] may_delete+0x5e/0x135
>   [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
>   [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8020b26e>] tracesys+0x71/0xda
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> input_log     D 00000000ffffffff     0  8004   7997
>   ffff81002d419ad0 0000000000000086 ffffffff8028fce8 ffff81007fb8de90
>   ffffffff80600cc0 0000000000000100 ffff81027f76aa80 ffff81007fb8de90
>   ffff8101fff32160 ffff81007fb8e0c8 000000012d419f50 ffff81007fb8e0c8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> output_log    D ffff81007e8b3cf8     0  8005   7997
>   ffff81002d4fdad0 0000000000000086 ffffffff8028fce8 ffff81007fbbc2c0
>   ffffffff80600cc0 0000000000000100 ffff81017ff649c0 ffff81007fbbc2c0
>   ffff81001ce46420 ffff81007fbbc4f8 000000012d4fdf50 ffff81007fbbc4f8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> production_lo D ffff81007e8b3cf8     0  8007   7997
>   ffff81002d615ad0 0000000000000082 ffffffff8028fce8 ffff81007d939bd0
>   ffffffff80600cc0 0000000000000100 ffff81027f76a3c0 ffff81007d939bd0
>   ffff81007ed50590 ffff81007d939e08 000000012d615f50 ffff81007d939e08
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8026f499>] mmap_region+0x41a/0x505
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> distribution_ D 00000000ffffffff     0  8008   7997
>   ffff81002d53fad0 0000000000000086 ffffffff8028fce8 ffff81007d93ac80
>   ffffffff80600cc0 0000000000000100 ffff81007f300600 ffff81007d93ac80
>   ffff8101fff33210 ffff81007d93aeb8 000000022d53ff50 ffff81007d93aeb8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> dir_check     D ffff81007e8b1690     0  8011   8009
>   ffff8101e1359d70 0000000000000082 ffff8101e1359d38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec10b0
>   ffff81007fb8de90 ffff8101bbec12e8 000000018028c917 ffff8101bbec12e8
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> afd_stat      D ffff81007e8b3cf8     0  8012   7997
>   ffff81002d69bb30 0000000000000082 ffff81002d69bda8 ffff81002d69bda8
>   ffffffff80600cc0 ffff81002d69bda8 ffffffff802a38d5 ffff81007d93c2c0
>   ffff81027eccd900 ffff81007d93c4f8 000000008023e66b ffff81007d93c4f8
> Call Trace:
>   [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80224a97>] __wake_up+0x38/0x4f
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
>   [<ffffffff802411a8>] hrtimer_start+0x100/0x122
>   [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> init_afd      D 00000000ffffffff     0  8020      1
>   ffff8101bbdd5b30 0000000000000086 ffff8101bbdd5dd0 ffff8101bbdd5da8
>   ffffffff80600cc0 ffff8101bbdd5db8 0000000000000010 ffff8101bbec1640
>   ffffffff8054d350 ffff8101bbec1878 0000000000000000 ffff8101bbec1878
> Call Trace:
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> receive_log   D 00000000ffffffff     0  8023   8020
>   ffff81027ee1bad0 0000000000000086 ffffffff8028fce8 ffff81027eccc2c0
>   ffffffff80600cc0 0000000000000100 ffff81007f300000 ffff81027eccc2c0
>   ffff8101fff33210 ffff81027eccc4f8 000000027ee1bf50 ffff81027eccc4f8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802ec6a2>] __ext4_journal_dirty_metadata+0x1e/0x46
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff8028ad96>] do_lookup+0x63/0x1a1
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> transfer_log  D 00000000ffffffff     0  8024   8020
>   ffff81027f799ad0 0000000000000086 ffffffff8028fce8 ffff81027eccc850
>   ffffffff80600cc0 0000000000000100 ffff81027d7443c0 ffff81027eccc850
>   ffffffff8054d350 ffff81027eccca88 000000007f799f50 ffff81027eccca88
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> archive_watch D 00000000ffffffff     0  8026   8020
>   ffff81027f7ddce0 0000000000000082 ffffffff802de7cd ffff810200000001
>   ffffffff80600cc0 ffff810000000001 ffffffff8027ce42 ffff81027ecc90b0
>   ffff8101fff33210 ffff81027ecc92e8 00000002802d9b7a ffff81027ecc92e8
> Call Trace:
>   [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
>   [<ffffffff8027ce42>] get_partial_node+0x15/0x78
>   [<ffffffff80294b15>] __d_lookup+0xbd/0x107
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8028c917>] __link_path_walk+0x6e0/0xdd4
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e3350>] ext4_unlink+0x3f/0x1fe
>   [<ffffffff8028b364>] may_delete+0x5e/0x135
>   [<ffffffff8028b9af>] vfs_unlink+0x5e/0xa3
>   [<ffffffff8028d985>] do_unlinkat+0xc1/0x142
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8020b26e>] tracesys+0x71/0xda
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> input_log     D ffff81007e8b3cf8     0  8027   8020
>   ffff81027eed1ad0 0000000000000082 ffffffff8028fce8 ffff81027ecca6f0
>   ffffffff80600cc0 0000000000000100 ffff81027d744e40 ffff81027ecca6f0
>   ffff81007fb8de90 ffff81027ecca928 000000017eed1f50 ffff81027ecca928
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> output_log    D ffff81007e8b3cf8     0  8028   8020
>   ffff81027d64dad0 0000000000000086 ffffffff8028fce8 ffff81027eccac80
>   ffffffff80600cc0 0000000000000100 ffff81010508c840 ffff81027eccac80
>   ffff81007ed574d0 ffff81027eccaeb8 000000027d64df50 ffff81027eccaeb8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> distribution_ D 00000000ffffffff     0  8031   8020
>   ffff81027ee1dad0 0000000000000086 ffffffff8028fce8 ffff81027ecc8590
>   ffffffff80600cc0 0000000000000100 ffff810105128a80 ffff81027ecc8590
>   ffff8101fff33210 ffff81027ecc87c8 000000027ee1df50 ffff81027ecc87c8
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8025c7ae>] generic_file_buffered_write+0x22c/0x658
>   [<ffffffff802dc663>] ext4_mark_inode_dirty+0x134/0x147
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80295a76>] touch_atime+0x12/0xfa
>   [<ffffffff80224b5b>] __wake_up_sync+0x3a/0x56
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> dir_check     D 00000000ffffffff     0  8034   8032
>   ffff81027ec01d70 0000000000000082 ffff81027ec01d38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81027eccb210
>   ffff8101fff33210 ffff81027eccb448 000000028028c917 ffff81027eccb448
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff8028cee0>] __link_path_walk+0xca9/0xdd4
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> afd_stat      D ffff81007e8b3cf8     0  8035   8020
>   ffff81027edebb30 0000000000000082 ffff81027edebda8 ffff81027edebda8
>   ffffffff80600cc0 ffff81027edebda8 ffffffff802a38d5 ffff81027eccd900
>   ffff81007ed57a60 ffff81027eccdb38 000000008023e66b ffff81027eccdb38
> Call Trace:
>   [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff802fe4bb>] do_get_write_access+0x3ab/0x3ed
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80224a97>] __wake_up+0x38/0x4f
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8020ac45>] do_notify_resume+0x81f/0x840
>   [<ffffffff802411a8>] hrtimer_start+0x100/0x122
>   [<ffffffff802258bb>] hrtick_start_fair+0x139/0x18a
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> fd            D ffff81007e8b3cf8     0  8037   8020
>   ffff81027ededb30 0000000000000086 ffff81027ededdd0 ffff81027ededda8
>   ffffffff80600cc0 ffff81027ededdb8 00000000000007e0 ffff81027eccde90
>   ffff81027ed937a0 ffff81027ecce0c8 0000000200000000 ffff81027ecce0c8
> Call Trace:
>   [<ffffffff80290dfd>] __pollwait+0x0/0xe3
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff8022f67f>] release_task+0x2cb/0x2dd
>   [<ffffffff80230175>] do_wait+0xae4/0xb88
>   [<ffffffff80286e19>] vfs_stat_fd+0x1b/0x4a
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> afd           D 00000000ffffffff     0 25970   2891
>   ffff81017fae7ad0 0000000000000086 ffffffff8028fce8 0000000000000246
>   ffffffff80600cc0 0000000000000100 ffff81020fe5dc00 ffff8101050ab210
>   ffff8101fff342c0 ffff8101050ab448 000000037fae7f50 ffff8101050ab448
> Call Trace:
>   [<ffffffff8028fce8>] poll_freewait+0x36/0x87
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80268f85>] __do_fault+0x321/0x369
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80237d9a>] group_send_sig_info+0x62/0x6f
>   [<ffffffff80237dd8>] kill_pid_info+0x31/0x3b
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b3cf8     0 26537   8037
>   ffff8101ff7b1b70 0000000000000086 ffffffff802411a8 000000000000148d
>   ffffffff80600cc0 0000000000000286 00000000ffffffff ffff8101bbec26f0
>   ffff81027eccb210 ffff8101bbec2928 00000002ffffffff ffff8101bbec2928
> Call Trace:
>   [<ffffffff802411a8>] hrtimer_start+0x100/0x122
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80466148>] thread_return+0x63/0xbb
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
>   [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
>   [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b1690     0 26543   8037
>   ffff8101ddec3d70 0000000000000086 ffff8101ddec3d38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff8101bbec3210
>   ffff81007ed542c0 ffff8101bbec3448 000000028028c917 ffff8101bbec3448
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b3cf8     0 26559   8037
>   ffff8101b11ddb60 0000000000000086 ffff8101bfc0b4d0 0000000000000202
>   ffffffff80600cc0 ffff8101bfc0b4d0 00007f1ea612f000 ffff8101bbec74d0
>   ffff81007ed574d0 ffff8101bbec7708 0000000200000000 ffff8101bbec7708
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80467779>] error_exit+0x0/0x51
>   [<ffffffff8025bae5>] find_lock_page+0x29/0x87
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
>   [<ffffffff8026d535>] unmap_region+0x10f/0x125
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D 00000000ffffffff     0 26569   8013
>   ffff8101f40c1b60 0000000000000082 ffff81013b896b00 0000000000000206
>   ffffffff80600cc0 ffff81013b896b00 00007f31838c7000 ffff8101bbec6f40
>   ffff8101fff32160 ffff8101bbec7178 0000000100000000 ffff8101bbec7178
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80467779>] error_exit+0x0/0x51
>   [<ffffffff8025bae5>] find_lock_page+0x29/0x87
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
>   [<ffffffff8026d535>] unmap_region+0x10f/0x125
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b3cf8     0 26583   8013
>   ffff8101bbf9fb60 0000000000000086 ffff81010c703160 0000000000000202
>   ffffffff80600cc0 ffff81010c703160 00007ffac6bd2000 ffff8101bbec6420
>   ffff81007d93fa60 ffff8101bbec6658 0000000100000000 ffff8101bbec6658
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80467779>] error_exit+0x0/0x51
>   [<ffffffff8025bae5>] find_lock_page+0x29/0x87
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
>   [<ffffffff8026d535>] unmap_region+0x10f/0x125
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D ffff810208bf59f8     0 26589  26567
>   ffff810208bf59d8 0000000000000082 ffff810163233090 ffffffff802e9b05
>   ffffffff80600cc0 ffffffff00058011 ffff810208bf5af8 ffff81027ecce420
>   ffff81001ce40000 ffff81027ecce658 0000000208bf5b58 ffff81027ecce658
> Call Trace:
>   [<ffffffff802e9b05>] ext4_ext_find_extent+0x67/0x26f
>   [<ffffffff80224a97>] __wake_up+0x38/0x4f
>   [<ffffffff803023b0>] jbd2_log_wait_commit+0xb6/0x108
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80300c3c>] jbd2_log_do_checkpoint+0x12d/0x3bd
>   [<ffffffff802a38d5>] __find_get_block_slow+0xcf/0xdb
>   [<ffffffff802a3c4d>] __find_get_block+0x14c/0x15c
>   [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
>   [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
>   [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
>   [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
>   [<ffffffff802e1b3a>] ext4_find_entry+0x293/0x592
>   [<ffffffff80300f53>] __jbd2_log_wait_for_space+0x87/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8029535a>] d_alloc+0x1f/0x189
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e24bd>] ext4_create+0x4c/0x105
>   [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
>   [<ffffffff8028b805>] vfs_create+0x7d/0xed
>   [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
>   [<ffffffff8028290c>] do_sys_open+0x46/0xc3
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D ffff81007e8b3cf8     0 26595  26584
>   ffff8101b1307ad0 0000000000000086 ffffffff8042530e ffff81014256f5d0
>   ffffffff80600cc0 ffff81017fa8fa00 ffffffff80242fdc ffff8101ff6d8b20
>   ffff81001ce46420 ffff8101ff6d8d58 000000018024153d ffff8101ff6d8d58
> Call Trace:
>   [<ffffffff8042530e>] nf_iterate+0x41/0x7d
>   [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
>   [<ffffffff8039d0b6>] loopback_xmit+0x80/0x86
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D ffff81007e8b3cf8     0 26602  26590
>   ffff81025dcf1c60 0000000000000082 ffffffff802e6fb3 0000000000000020
>   ffffffff80600cc0 ffff81014256f090 00000000000300aa ffff81027ed942c0
>   ffff81007d93e420 ffff81027ed944f8 000000017ae2b000 ffff81027ed944f8
> Call Trace:
>   [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
>   [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e24bd>] ext4_create+0x4c/0x105
>   [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
>   [<ffffffff8028b805>] vfs_create+0x7d/0xed
>   [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
>   [<ffffffff8028290c>] do_sys_open+0x46/0xc3
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D 00000000ffffffff     0 26605   8013
>   ffff81001cdc9b60 0000000000000086 ffff8101bfc0b4d0 0000000000000216
>   ffffffff80600cc0 ffff8101bfc0b4d0 00007f628a734000 ffff81001ce43d30
>   ffff8101fff33210 ffff81001ce43f68 0000000200000000 ffff81001ce43f68
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80467779>] error_exit+0x0/0x51
>   [<ffffffff8025bae5>] find_lock_page+0x29/0x87
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
>   [<ffffffff8026d535>] unmap_region+0x10f/0x125
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b1690     0 26608   8013
>   ffff81003eb4fd70 0000000000000082 ffff81003eb4fd38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce43210
>   ffff81001ce410b0 ffff81001ce43448 000000018028c917 ffff81001ce43448
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b1690     0 26609   8013
>   ffff81003eb63d70 0000000000000082 ffff81003eb63d38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce442c0
>   ffff81017faea160 ffff81001ce444f8 000000008028c917 ffff81001ce444f8
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b1690     0 26610   8013
>   ffff81003e98dd70 0000000000000082 ffff81003e98dd38 ffff81007edfcb00
>   ffffffff80600cc0 0000000000000101 ffffffff80293dd5 ffff81001ce42c80
>   ffff81017fae8590 ffff81001ce42eb8 000000028028c917 ffff81001ce42eb8
> Call Trace:
>   [<ffffffff80293dd5>] dput+0x21/0x10a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028ac53>] lock_rename+0x35/0xc5
>   [<ffffffff8028d545>] do_rename+0x98/0x1b2
>   [<ffffffff802921db>] __posix_lock_file+0x43d/0x44e
>   [<ffffffff8027ce42>] get_partial_node+0x15/0x78
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D 00000000ffffffff     0 26611   8013
>   ffff81001ced3b60 0000000000000082 ffff81002d7fa0d0 ffff81002d7fa0d0
>   ffffffff80600cc0 00000000486202ae 0000000210e7c865 ffff81001ce41640
>   ffffffff8054d350 ffff81001ce41878 0000000000000003 ffff81001ce41878
> Call Trace:
>   [<ffffffff80411b14>] dev_queue_xmit+0x21a/0x232
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff80282afc>] __dentry_open+0x141/0x226
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D 00000000ffffffff     0 26613   8013
>   ffff8101e1321b30 0000000000000082 0000000000000296 00000000662df001
>   ffffffff80600cc0 ffff8101d40c9e90 ffffffff80242fdc ffff8101ff6dc850
>   ffff8101fff33210 ffff8101ff6dca88 000000028024153d ffff8101ff6dca88
> Call Trace:
>   [<ffffffff80242fdc>] getnstimeofday+0x38/0x92
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffffa0070d34>] :nf_conntrack:nf_ct_deliver_cached_events+0x4d/0x7d
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff80269428>] do_wp_page+0x45b/0x4ba
>   [<ffffffff8026aaed>] handle_mm_fault+0x62d/0x6a3
>   [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
>   [<ffffffff80286f41>] cp_new_stat+0xe9/0xfc
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff80467779>] error_exit+0x0/0x51
> 
> vsftpd        D ffff81007e8b3cf8     0 26614  26606
>   ffff8100588b1c90 0000000000000086 ffffffff802411a8 000000000000148d
>   ffffffff80600cc0 0000000000000286 00000000ffffffff ffff81001ce40000
>   ffff8101ff6dd900 ffff81001ce40238 00000002ffffffff ffff81001ce40238
> Call Trace:
>   [<ffffffff802411a8>] hrtimer_start+0x100/0x122
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80466148>] thread_return+0x63/0xbb
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80296e64>] inode_setattr+0x103/0x10a
>   [<ffffffff802dc9b0>] ext4_setattr+0x1eb/0x244
>   [<ffffffff80296fbb>] notify_change+0x150/0x26d
>   [<ffffffff80283568>] do_truncate+0x5e/0x79
>   [<ffffffff80256c42>] audit_syscall_entry+0x11f/0x152
>   [<ffffffff80283664>] sys_ftruncate+0xe1/0xfd
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> bzip2         D 00000000ffffffff     0 26616  26615
>   ffff8102043d9ad0 0000000000000082 ffffffff80278233 ffff810256ebc660
>   ffffffff80600cc0 00007f5eba717000 0000000000000002 ffff81027ed91640
>   ffff8101fff342c0 ffff81027ed91878 000000038026a789 ffff81027ed91878
> Call Trace:
>   [<ffffffff80278233>] alloc_page_vma+0x154/0x221
>   [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
>   [<ffffffff80266f91>] zone_statistics+0x3c/0x90
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80268f85>] __do_fault+0x321/0x369
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a745>] hrtick_set+0xa1/0x10a
>   [<ffffffff80466148>] thread_return+0x63/0xbb
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> sf_ftp        D ffff81007e8b3cf8     0 26621   8037
>   ffff8101b1103b60 0000000000000082 ffff81013b896b00 0000000000000206
>   ffffffff80600cc0 ffff81013b896b00 00007f4f1e28e000 ffff8101ff6df4d0
>   ffff8101bbec6f40 ffff8101ff6df708 0000000100000000 ffff8101ff6df708
> Call Trace:
>   [<ffffffff80224f1d>] default_wake_function+0x0/0xe
>   [<ffffffff8042e751>] ip_queue_xmit+0x2a2/0x2f8
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80467779>] error_exit+0x0/0x51
>   [<ffffffff8025bae5>] find_lock_page+0x29/0x87
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8028a28d>] pipe_write+0x4bc/0x4ce
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff80273714>] free_pages_and_swap_cache+0x70/0x8b
>   [<ffffffff8026d535>] unmap_region+0x10f/0x125
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D ffff81007e8b3cf8     0 26625  26617
>   ffff810039cf9b70 0000000000000082 ffff810270718a80 ffff81022646e2a0
>   ffffffff80600cc0 ffff810039cf9d44 0000000000000000 ffff81001ce469b0
>   ffff81001ce41640 ffff81001ce46be8 00000000802a3c7b ffff81001ce46be8
> Call Trace:
>   [<ffffffff802de7cd>] ext4_getblk+0xb4/0x170
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80294b15>] __d_lookup+0xbd/0x107
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e2b08>] ext4_rename+0x4d/0x64a
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8028bc5c>] vfs_rename+0x1fd/0x376
>   [<ffffffff8028d5fe>] do_rename+0x151/0x1b2
>   [<ffffffff802614c3>] __alloc_pages_internal+0xd2/0x40c
>   [<ffffffff80256944>] audit_syscall_exit+0x2ee/0x30c
>   [<ffffffff8028d6ab>] sys_renameat+0x4c/0x6b
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D 00000000ffffffff     0 26632  26626
>   ffff8101dddc3c60 0000000000000082 ffffffff802e6fb3 0000000000000020
>   ffffffff80600cc0 ffff8101d40c8610 0000000000031014 ffff8101ff6dd900
>   ffff8101fff33210 ffff8101ff6ddb38 000000021d4e6000 ffff8101ff6ddb38
> Call Trace:
>   [<ffffffff802e6fb3>] ext4fs_dirhash+0x10d/0x1d2
>   [<ffffffff802e04cf>] dx_probe+0xa1/0x29e
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff8029925e>] mntput_no_expire+0x27/0x11e
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802e24bd>] ext4_create+0x4c/0x105
>   [<ffffffff802e27c3>] ext4_lookup+0x97/0xc1
>   [<ffffffff8028b805>] vfs_create+0x7d/0xed
>   [<ffffffff8028e3ac>] do_filp_open+0x1ed/0x832
>   [<ffffffff8028290c>] do_sys_open+0x46/0xc3
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> vsftpd        D 00000000ffffffff     0 26635  26633
>   ffff8100066c9dd0 0000000000000086 ffffffff804082ac ffff8100066c9d34
>   ffffffff80600cc0 0000000e9aa3566f ffff81007edfcbc8 ffff81007d93e420
>   ffff8101fff32160 ffff81007d93e658 0000000100000000 ffff81007d93e658
> Call Trace:
>   [<ffffffff804082ac>] sock_common_recvmsg+0x30/0x45
>   [<ffffffff8028d0ba>] path_walk+0xaf/0xb9
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff8028e2cb>] do_filp_open+0x10c/0x832
>   [<ffffffff80282837>] get_unused_fd_flags+0x7f/0x10e
>   [<ffffffff8028290c>] do_sys_open+0x46/0xc3
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> bzip2         D 00000000ffffffff     0 26638  26637
>   ffff8101427cdad0 0000000000000082 ffffffff80278233 ffff8101f6fa70c0
>   ffffffff80600cc0 00007f0c65f4e000 0000000000000002 ffff81017faec850
>   ffffffff8054d350 ffff81017faeca88 000000008026a789 ffff81017faeca88
> Call Trace:
>   [<ffffffff80278233>] alloc_page_vma+0x154/0x221
>   [<ffffffff802a3c7b>] __getblk+0x1e/0x28d
>   [<ffffffff804666c5>] __mutex_lock_slowpath+0x6a/0xa0
>   [<ffffffff80466561>] mutex_lock+0xa/0xb
>   [<ffffffff80300f1e>] __jbd2_log_wait_for_space+0x52/0xac
>   [<ffffffff802feac7>] start_this_handle+0x2eb/0x370
>   [<ffffffff80260c63>] rmqueue_bulk+0x42/0x8a
>   [<ffffffff802fecae>] jbd2_journal_start+0x9a/0xce
>   [<ffffffff802dc772>] ext4_dirty_inode+0x28/0x7b
>   [<ffffffff802a00fe>] __mark_inode_dirty+0x29/0x187
>   [<ffffffff80295a1f>] file_update_time+0xaf/0xf4
>   [<ffffffff8025d031>] __generic_file_aio_write_nolock+0x265/0x37e
>   [<ffffffff80268f85>] __do_fault+0x321/0x369
>   [<ffffffff8025d815>] generic_file_aio_write+0x64/0xc4
>   [<ffffffff802d9d8c>] ext4_file_write+0x96/0x11a
>   [<ffffffff80283eb3>] do_sync_write+0xce/0x113
>   [<ffffffff8021ed48>] do_page_fault+0x471/0x825
>   [<ffffffff8023e74a>] autoremove_wake_function+0x0/0x2e
>   [<ffffffff8022a72f>] hrtick_set+0x8b/0x10a
>   [<ffffffff80466148>] thread_return+0x63/0xbb
>   [<ffffffff802846d8>] vfs_write+0xad/0x136
>   [<ffffffff8028481d>] sys_write+0x45/0x6e
>   [<ffffffff8020b2d2>] tracesys+0xd5/0xda
> 
> I used ext4-patch-queue-b5db22ef52ed53d8e3fa978a5a29e1609c9333aa.tar.gz and
> a clean linux-2.6.26-rc6 with no other patches.
> 
> Holger
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance of ext4

[Aneesh Kumar K.V]

On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> 
> On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > On Tue, 24 Jun 2008, Mingming wrote:
> > 
> > >
> > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > >> Yes, with this patch applied on top of latest patch queue I no longer
> > >> get truncated files, after running a short test. Tomorrow I will do some
> > >> more thorough testing and use the patch you have send to me in a separate
> > >> mail. The above patch did not apply but it was easy to apply by hand.
> > >
> > >
> > > Thanks for quick response and test. I have updated the patch queue with
> > > above patch merged. Please let me know if you still see apply issue and
> > > file size update issue with current patch queue.
> > >
> > Thanks, it applies without any problems. However I still hit an oops. What
> > I find strange is that I got the oops just as the benchmark is done and
> > all process where shutting down. The same behaviour I reported here:
> > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > and HW Raid). Anyway, here the dmesg output:
> > 
> > kjournald2 starting.  Commit interval 15 seconds
> > EXT4 FS on md7, internal journal
> > EXT4-fs: mounted filesystem with ordered data mode.
> > EXT4-fs: file extents enabled
> > EXT4-fs: mballoc enabled
> > JBD: barrier-based sync failed on md7 - disabling barriers
> > ------------[ cut here ]------------
> > kernel BUG at fs/ext4/inode.c:1667!
> 
> Did not get a chance to look more closely today, but it's point to this
> code in ext4_da_writepage()
> 
>  page_bufs = page_buffers(page);
> 
> and appearently it's BUG_ON at 
>  BUG_ON(!PagePrivate(page)); in page_buffers().
> 
> 

Ok so we are doing the journal_commit and meanwhile shrink_page_list
dropped the buffer. I guess what is happening is


journal_submit_inode_data_buffers
generic_writepages
write_cache_pages
pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
foreach(page)
							shrink_page_list
							lock_page
							ext4_releasepage
							try_to_free_buffers
							drop_buffers
							cancel_dirty_page
							unlock_page
lock_page()
BUG_ON(!PagePrivate(page));


How about the below ? I am not sure journalled mode writepage would need a
simillar change. We don't need to redirty the page because
that would mean the next journal_commit will hit the BUG_ON.
If we need to do a BUG_ON

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index fd67b34..94e5ab0 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1662,8 +1662,17 @@ static int ext4_da_writepage(struct page *page,
 		 * writing the page in case we would have to do so.
 		 * We reach here also via journal_submit_inode_data_buffers
 		 */
+		if (!page_has_buffers(page)) {
+			/*
+			 * This can happen when we are called via
+			 * journal_submit_inode_data_buffers and
+			 * shrink_page_list parallely did drop_buffers
+			 * after write_cache_pages did a pagevec_lookup_tag
+			 */
+			unlock_page(page);
+			return 0;
+		}
 		size = i_size_read(inode);

Re: Performance of ext4

[Aneesh Kumar K.V]

On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> > 
> > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > On Tue, 24 Jun 2008, Mingming wrote:
> > > 
> > > >
> > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > >> more thorough testing and use the patch you have send to me in a separate
> > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > >
> > > >
> > > > Thanks for quick response and test. I have updated the patch queue with
> > > > above patch merged. Please let me know if you still see apply issue and
> > > > file size update issue with current patch queue.
> > > >
> > > Thanks, it applies without any problems. However I still hit an oops. What
> > > I find strange is that I got the oops just as the benchmark is done and
> > > all process where shutting down. The same behaviour I reported here:
> > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > and HW Raid). Anyway, here the dmesg output:
> > > 
> > > kjournald2 starting.  Commit interval 15 seconds
> > > EXT4 FS on md7, internal journal
> > > EXT4-fs: mounted filesystem with ordered data mode.
> > > EXT4-fs: file extents enabled
> > > EXT4-fs: mballoc enabled
> > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > ------------[ cut here ]------------
> > > kernel BUG at fs/ext4/inode.c:1667!
> > 
> > Did not get a chance to look more closely today, but it's point to this
> > code in ext4_da_writepage()
> > 
> >  page_bufs = page_buffers(page);
> > 
> > and appearently it's BUG_ON at 
> >  BUG_ON(!PagePrivate(page)); in page_buffers().
> > 
> > 
> 
> Ok so we are doing the journal_commit and meanwhile shrink_page_list
> dropped the buffer. I guess what is happening is
> 
> 
> journal_submit_inode_data_buffers
> generic_writepages
> write_cache_pages
> pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
> foreach(page)
> 							shrink_page_list
> 							lock_page
> 							ext4_releasepage
> 							try_to_free_buffers
> 							drop_buffers
> 							cancel_dirty_page
> 							unlock_page
> lock_page()
> BUG_ON(!PagePrivate(page));
> 
> 
> How about the below ? 

or update write_cache_pages not to call writepage if the page is not
dirty ?


diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index ded57d5..0a13702 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -929,6 +929,11 @@ int write_cache_pages(struct address_space *mapping,
 				continue;
 			}
 
+			if (!PageDirty(page)) {
+				unlock_page(page);
+				continue;
+			}
+
 			ret = (*writepage)(page, wbc, data);
 
 			if (unlikely(ret == AOP_WRITEPAGE_ACTIVATE)) {

Re: Performance of ext4

[Jan Kara]

On Fri 27-06-08 15:19:13, Aneesh Kumar K.V wrote:
> On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> > On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> > > 
> > > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > > On Tue, 24 Jun 2008, Mingming wrote:
> > > > 
> > > > >
> > > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > > >> more thorough testing and use the patch you have send to me in a separate
> > > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > > >
> > > > >
> > > > > Thanks for quick response and test. I have updated the patch queue with
> > > > > above patch merged. Please let me know if you still see apply issue and
> > > > > file size update issue with current patch queue.
> > > > >
> > > > Thanks, it applies without any problems. However I still hit an oops. What
> > > > I find strange is that I got the oops just as the benchmark is done and
> > > > all process where shutting down. The same behaviour I reported here:
> > > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > > and HW Raid). Anyway, here the dmesg output:
> > > > 
> > > > kjournald2 starting.  Commit interval 15 seconds
> > > > EXT4 FS on md7, internal journal
> > > > EXT4-fs: mounted filesystem with ordered data mode.
> > > > EXT4-fs: file extents enabled
> > > > EXT4-fs: mballoc enabled
> > > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > > ------------[ cut here ]------------
> > > > kernel BUG at fs/ext4/inode.c:1667!
> > > 
> > > Did not get a chance to look more closely today, but it's point to this
> > > code in ext4_da_writepage()
> > > 
> > >  page_bufs = page_buffers(page);
> > > 
> > > and appearently it's BUG_ON at 
> > >  BUG_ON(!PagePrivate(page)); in page_buffers().
> > > 
> > > 
> > 
> > Ok so we are doing the journal_commit and meanwhile shrink_page_list
> > dropped the buffer. I guess what is happening is
> > 
> > 
> > journal_submit_inode_data_buffers
> > generic_writepages
> > write_cache_pages
> > pagevec_lookup_tag(..PAGECACHE_TAG_DIRTY,..)
> > foreach(page)
> > 							shrink_page_list
> > 							lock_page
> > 							ext4_releasepage
> > 							try_to_free_buffers
> > 							drop_buffers
> > 							cancel_dirty_page
> > 							unlock_page
> > lock_page()
> > BUG_ON(!PagePrivate(page));
> > 
> > 
> > How about the below ? 
> 
> or update write_cache_pages not to call writepage if the page is not
> dirty ?
  But that is already happening :) Look a few lines above your patch into
clear_page_dirty_for_io()...

									Honza

> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index ded57d5..0a13702 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -929,6 +929,11 @@ int write_cache_pages(struct address_space *mapping,
>  				continue;
>  			}
>  
> +			if (!PageDirty(page)) {
> +				unlock_page(page);
> +				continue;
> +			}
> +
>  			ret = (*writepage)(page, wbc, data);
>  
>  			if (unlikely(ret == AOP_WRITEPAGE_ACTIVATE)) {
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Performance of ext4

[Aneesh Kumar K.V]

On Fri, Jun 27, 2008 at 12:00:24PM +0200, Jan Kara wrote:
> On Fri 27-06-08 15:19:13, Aneesh Kumar K.V wrote:
> > On Fri, Jun 27, 2008 at 02:44:59PM +0530, Aneesh Kumar K.V wrote:
> > > On Wed, Jun 25, 2008 at 05:46:39PM -0700, Mingming wrote:
> > > > 
> > > > On Wed, 2008-06-25 at 09:09 +0000, Holger Kiehl wrote:
> > > > > On Tue, 24 Jun 2008, Mingming wrote:
> > > > > 
> > > > > >
> > > > > > On Tue, 2008-06-24 at 21:12 +0000, Holger Kiehl wrote:
> > > > > >> Yes, with this patch applied on top of latest patch queue I no longer
> > > > > >> get truncated files, after running a short test. Tomorrow I will do some
> > > > > >> more thorough testing and use the patch you have send to me in a separate
> > > > > >> mail. The above patch did not apply but it was easy to apply by hand.
> > > > > >
> > > > > >
> > > > > > Thanks for quick response and test. I have updated the patch queue with
> > > > > > above patch merged. Please let me know if you still see apply issue and
> > > > > > file size update issue with current patch queue.
> > > > > >
> > > > > Thanks, it applies without any problems. However I still hit an oops. What
> > > > > I find strange is that I got the oops just as the benchmark is done and
> > > > > all process where shutting down. The same behaviour I reported here:
> > > > > http://www.ussg.iu.edu/hypermail/linux/kernel/0806.2/2113.html
> > > > > Only this time I got just one oops. This is on x86_64 system (4 Opteron CPU's
> > > > > and SW Raid 1+0). I have not seen this on my home system x86 (1 Dual Core
> > > > > and HW Raid). Anyway, here the dmesg output:
> > > > > 
> > > > > kjournald2 starting.  Commit interval 15 seconds
> > > > > EXT4 FS on md7, internal journal
> > > > > EXT4-fs: mounted filesystem with ordered data mode.
> > > > > EXT4-fs: file extents enabled
> > > > > EXT4-fs: mballoc enabled
> > > > > JBD: barrier-based sync failed on md7 - disabling barriers
> > > > > ------------[ cut here ]------------
> > > > > kernel BUG at fs/ext4/inode.c:1667!
> > > > 
> > > > Did not get a chance to look more closely today, but it's point to this
> > > > code in ext4_da_writepage()
> > > > 
> > > >  page_bufs = page_buffers(page);
> > > > 
> > > > and appearently it's BUG_ON at 
> > > >  BUG_ON(!PagePrivate(page)); in page_buffers().
> > > > 
> > > > 
> > > 

How about this ?

commit 174d555d8effb480a23d5dea8db698d1bc2cfa7d
Author: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Date:   Fri Jun 27 23:04:28 2008 +0530

    ext4: call ext4_page_mkwrite even for MappedToDisk pages
    
    We can have pages that are fully mapped to disk. The
    mappedtodisk flag is used to indicate that every
    buffer_head in the page have a mapping block allocated
    on disk. But that doesn't gurantee that we have initialized
    those buffer_head and added it to page via page->private.
    This causes writepage to go BUGON when it find a page
    that have NULL page->private.
    
    The fix is to make sure we initialize the buffer_head and add
    it to page when we are going to write to the page. This can
    be done via ext4_page_mkwrite
    
    Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>

diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 10f1d5d..11ebe88 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -3978,8 +3978,6 @@ int ext4_page_mkwrite(struct vm_area_struct *vma, struct page *page)
 		goto out_unlock;
 	}
 	ret = 0;
-	if (PageMappedToDisk(page))
-		goto out_unlock;
 
 	if (page->index == size >> PAGE_CACHE_SHIFT)
 		len = size & ~PAGE_CACHE_MASK;

Re: Performance of ext4

[Mingming]

On Tue, 2008-06-24 at 08:37 +0530, Aneesh Kumar K.V wrote:
> On Mon, Jun 23, 2008 at 05:31:32PM -0700, Mingming wrote:
> > 
> > On Mon, 2008-06-23 at 23:15 +0530, Aneesh Kumar K.V wrote:
> > 
> > > I found one place where we fail to update i_disksize. Can you try this
> > > patch ?
> > > 
> > > diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> > > index 33f940b..9fa737f 100644
> > > --- a/fs/ext4/inode.c
> > > +++ b/fs/ext4/inode.c
> > > @@ -1620,7 +1620,10 @@ static int ext4_da_writepage(struct page *page,
> > >  	loff_t size;
> > >  	unsigned long len;
> > >  	handle_t *handle = NULL;
> > > +	ext4_lblk_t block;
> > > +	loff_t disksize;
> > >  	struct buffer_head *page_bufs;
> > > +	struct buffer_head *bh, *head;
> > >  	struct inode *inode = page->mapping->host;
> > > 
> > >  	handle = ext4_journal_current_handle();
> > > @@ -1662,6 +1665,38 @@ static int ext4_da_writepage(struct page *page,
> > >  	else
> > >  		ret = block_write_full_page(page, ext4_da_get_block_write, wbc);
> > > 
> > > +	 if (ret)
> > > +		 return ret;
> > > +	/*
> > > +	 * When called via shrink_page_list and if we don't have any unmapped
> > > +	 * buffer_head we still could have written some new content in an
> > > +	 * already mapped buffer. That means we need to extent i_disksize here
> > > +	 */
> > 
> > In this case(when extend the file without need block allocation),
> > wouldn't make sense to update the i_disksize at write_end() time? So
> > that the window of i_size different from i_disksize could be much
> > smaller in this case.
> > 
> > 
> > Something like below? (untested)
> 
> In this case you will have to start a transaction in write_begin . With
> the below code transaction is started inside page_lock. Also I don't
> think we need needed_blocks credit just 1 should be enough because we
> are not doing any block allocation here. We just need to update the
> inode block.
> 
> 

Right! I will update the patch and merge it to the patch queue.

Mingming
> -aneesh
> 
> > 
> > Signed-off-by: Mingming Cao <cmm@us.ibm.com>
> > ---
> >  fs/ext4/inode.c |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++------
> >  1 file changed, 60 insertions(+), 6 deletions(-)
> > 
> > Index: linux-2.6.26-rc6/fs/ext4/inode.c
> > ===================================================================
> > --- linux-2.6.26-rc6.orig/fs/ext4/inode.c	2008-06-23 17:28:01.000000000 -0700
> > +++ linux-2.6.26-rc6/fs/ext4/inode.c	2008-06-23 17:28:10.000000000 -0700
> > @@ -1573,6 +1573,65 @@ static int ext4_da_write_begin(struct fi
> >  	return ret;
> >  }
> > 
> > +static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> > +{
> > +	return !buffer_mapped(bh) || buffer_delay(bh);
> > +}
> > +
> > +static int ext4_da_write_end(struct file *file,
> > +				struct address_space *mapping,
> > +				loff_t pos, unsigned len, unsigned copied,
> > +				struct page *page, void *fsdata)
> > +{
> > +	handle_t *handle = NULL;
> > +	struct inode *inode = mapping->host;
> > +	int needed_blocks = ext4_writepage_trans_blocks(inode);
> > +	unsigned from, to;
> > +	int ret = 0, ret2;
> > +
> > +	from = pos & (PAGE_CACHE_SIZE - 1);
> > +	to = from + len;
> > +
> > +	if (ret == 0) {
> > +		/*
> > +		 * generic_write_end() will run mark_inode_dirty() if i_size
> > +		 * changes.  So let's piggyback the i_disksize mark_inode_dirty
> > +		 * into that.
> > +		 */
> > +		loff_t new_i_size;
> > +
> > +		new_i_size = pos + copied;
> > +		if (new_i_size > EXT4_I(inode)->i_disksize)
> > +			if (!walk_page_buffers(NULL, page_buffers(page),
> > +					       0, len, NULL, ext4_bh_unmapped_or_delay)){
> > +				/*
> > +				 * Updating i_disksize when extending file without
> > +				 * need block allocation
> > +				 */
> > +        			handle = ext4_journal_start(inode, needed_blocks);
> > +				if (IS_ERR(handle)) {
> > +			                ret = PTR_ERR(handle);
> > +			                return ret;
> > +				}
> > +				if (ext4_should_order_data(inode))
> > +					ret = ext4_jbd2_file_inode(handle, inode);
> > +
> > +				EXT4_I(inode)->i_disksize = new_i_size;
> > +			}
> > +		ret2 = generic_write_end(file, mapping, pos, len, copied,
> > +							page, fsdata);
> > +		copied = ret2;
> > +		if (ret2 < 0)
> > +			ret = ret2;
> > +	}
> > +	if (handle)
> > +		ret2 = ext4_journal_stop(handle);
> > +	if (!ret)
> > +		ret = ret2;
> > +
> > +	return ret ? ret : copied;
> > +}
> > +
> >  static void ext4_da_invalidatepage(struct page *page, unsigned long offset)
> >  {
> >  	struct buffer_head *head, *bh;
> > @@ -1682,11 +1741,6 @@ static int bput_one(handle_t *handle, st
> >  	return 0;
> >  }
> > 
> > -static int ext4_bh_unmapped_or_delay(handle_t *handle, struct buffer_head *bh)
> > -{
> > -	return !buffer_mapped(bh) || buffer_delay(bh);
> > -}
> > -
> >  /*
> >   * Note that we don't need to start a transaction unless we're journaling data
> >   * because we should have holes filled from ext4_page_mkwrite(). We even don't
> > @@ -2050,7 +2104,7 @@ static const struct address_space_operat
> >  	.writepages	= ext4_da_writepages,
> >  	.sync_page	= block_sync_page,
> >  	.write_begin	= ext4_da_write_begin,
> > -	.write_end	= generic_write_end,
> > +	.write_end	= ext4_da_write_end,
> >  	.bmap		= ext4_bmap,
> >  	.invalidatepage	= ext4_da_invalidatepage,
> >  	.releasepage	= ext4_releasepage,
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Performance of ext4

[Holger Kiehl]

On Mon, 23 Jun 2008, Aneesh Kumar K.V wrote:

> On Fri, Jun 20, 2008 at 09:21:48AM +0000, Holger Kiehl wrote:
>> On Fri, 20 Jun 2008, Theodore Tso wrote:
>>
>>> On Fri, Jun 20, 2008 at 08:32:52AM +0000, Holger Kiehl wrote:
>>>>> It sounds like i_size is actually dropping in
>>>>> size at some pointer long after the file was written.  If I had to
>>>
>>> sorry, "at some point"...
>>>
>>>>> guess the value in the inode cache is correct; and perhaps so is the
>>>>> value on the journal.  But somehow, the wrong value is getting written
>>>>> to disk
>>>
>>> Or, "the right value is never getting written to disk".  (Which as I
>>> think about it is more likely; it's likely that an update to i_size is
>>> getting *lost*, perhaps because the delalloc code is possibly
>>> modifying i_size without starting a transaction first.  Again this is
>>> just a guess.)
>>>
>>>> What I find strange is that the missing parts of the file are not for
>>>> example exactly 512 or 1024 or 4096 bytes it is mostly some odd number
>>>> of bytes.
>>>
>>> Is there any chance the truncation point is related to how the program
>>> is writing its output file?  i.e., if it is a text file, is the
>>> truncation happening after a new-line or when the stdio library might
>>> have done an explicit or implicit fflush()?
>>>
>> When the benchmark runs it writes to stdout and with tee to the result
>> file. It first writes some information about the system, prepares the
>> test files (creates lots of small files), calls sync and then starts
>> the test. Then every minute one line gets written to the result file.
>> Often I have seen that everything after the sync was missing. But
>> sometimes it happened that some parts at the end are missing. But it
>> was always a clean cut, that is there where no lines that where cut
>> partially. The lines where always complete.
>>
>
> I found one place where we fail to update i_disksize. Can you try this
> patch ?
>
Yes, I would like to however when I take
ext4-patch-queue-70acdb9605784bd5c4b06e1a19761828a494a337.tar.gz (which is
the current ext4-patch-queue from http://repo.or.cz/w/ext4-patch-queue.git)
and apply those to linux-2.6.26-rc6 I get the following reject:


***************
*** 574,579 ****
   	INIT_LIST_HEAD(&ei->i_prealloc_list);
   	spin_lock_init(&ei->i_prealloc_lock);
   	jbd2_journal_init_jbd_inode(&ei->jinode, &ei->vfs_inode);
   	return &ei->vfs_inode;
   }

--- 574,584 ----
   	INIT_LIST_HEAD(&ei->i_prealloc_list);
   	spin_lock_init(&ei->i_prealloc_lock);
   	jbd2_journal_init_jbd_inode(&ei->jinode, &ei->vfs_inode);
+ 	ei->i_reserved_data_blocks = 0;
+ 	ei->i_reserved_meta_blocks = 0;
+ 	ei->i_allocated_meta_blocks = 0;
+ 	ei->i_delalloc_reserved_flag = 0;
+ 	spin_lock_init(&(ei->i_block_reservation_lock));
   	return &ei->vfs_inode;
   }

Which is from delalloc-ext4-ENOSPC-handling.patch. What am I doing wrong?
I could apply this by hand but I do not know if this would be correct.
Please can anyone advice what I need to do?

Thanks,
Holger

Re: Performance of ext4

[Andreas Dilger]

On Jun 20, 2008  09:21 +0000, Holger Kiehl wrote:
> When the benchmark runs it writes to stdout and with tee to the result
> file. It first writes some information about the system, prepares the
> test files (creates lots of small files), calls sync and then starts
> the test. Then every minute one line gets written to the result file.
> Often I have seen that everything after the sync was missing. But
> sometimes it happened that some parts at the end are missing. But it
> was always a clean cut, that is there where no lines that where cut
> partially. The lines where always complete.

Could you enhance your test to record the file size from "stat -c %s {file}"
at the end of the test, and also "dumpe2fs -c -R 'stat <inum>' {dev}"
where <inum> is from "stat -c %i {file}".  If these two don't match after
60s, or after a "sync; sync" then there will likely be a data loss.

With delalloc it is expected that the "debugfs" output will not match up
to 30s+ after the last modification, because the write is only in memory.
With ext3 this window would be much smaller, and in fact not visible from
userspace because "debugfs" would be accessing the same (in memory) buffer
as the kernel, so it can't even access the stale data on disk.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: Performance of ext4

[Holger Kiehl]

On Thu, 19 Jun 2008, Theodore Tso wrote:

> On Tue, Jun 17, 2008 at 11:42:36AM +0000, Holger Kiehl wrote:
>> Note how the size of file results.24033.helena.dwd.de changes from
>> 9230 before the test to 8208 bytes after the test. Also note the
>> date both have the same timestamp "2008-06-17 04:35". I have made a
>> copy of results.24033.helena.dwd.de before the test and compared it
>> with that after the test. The file is just truncated by 1022 bytes
>> and there is no garbage.
>
> So the corruption is always a truncation, correct?
>
Correct.

> Did you notice the problem with ext4 w/o the patch queue?
>
No, without patch queue it I did not not notice the problem.

> I have a
> suspicion that the problem may have been introduced by the delayed
> allocation code, but I don't have hard evidence.  When you rerun your
> benchmark (which seems to be the closest thing we have to a
> reproduction case), it would be interesting to know if the problem
> goes away with -o nodelalloc (again, it would localize where we need
> to look).
>
Ok I will do thats soon as I have a system available.

Holger

Re: Performance of ext4

[Holger Kiehl]

On Fri, 20 Jun 2008, Holger Kiehl wrote:

> On Thu, 19 Jun 2008, Theodore Tso wrote:
>
>> I have a
>> suspicion that the problem may have been introduced by the delayed
>> allocation code, but I don't have hard evidence.  When you rerun your
>> benchmark (which seems to be the closest thing we have to a
>> reproduction case), it would be interesting to know if the problem
>> goes away with -o nodelalloc (again, it would localize where we need
>> to look).
>> 
> Ok I will do thats soon as I have a system available.
>
No, with nodelalloc the problem does not occur. I can now very reliably
repoduce the problem if I run the benchmark for 5 minutes and then unmount
the filesystem and remount it. Anything else I can do to find the problem?

During testing on a newly setup system (used the same kernel 2.6.26-rc5+ext4-
patch-queue) I have hit two oopses:

------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1931!
invalid opcode: 0000 [1] SMP 
CPU 3 
Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy ohci_hcd i2c_amd756 k8temp i2c_core button usbcore
Pid: 242, comm: pdflush Not tainted 2.6.26-rc5 #4
RIP: 0010:[<ffffffff802dd1c5>]  [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP: 0018:ffff81007f329c20  EFLAGS: 00010246
RAX: 0a0000000001002d RBX: ffffe20006f658c8 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff81007f329e50 RDI: ffffe20006f658c8
RBP: ffff81007f329e50 R08: ffff81027ff054c0 R09: ffff81007f329b60
R10: 0000000000000040 R11: 0000000000000040 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fbc99d8b6f0(0000) GS:ffff81027ff0e000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f64c0d43488 CR3: 00000001efa65000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process pdflush (pid: 242, threadinfo ffff81007f328000, task ffff81007fbc3d30)
Stack:  0000000000000000 ffff810149bc41d0 ffff81007f329e50 ffffffff8026179a
  ffffe20006f658c8 ffffffff80261f57 ffff810149bc41d0 ffffffff80261790
  ffff810149bc41d0 0000000180241154 ffff8101ff6caea0 ffffffffffffffff
Call Trace:
  [<ffffffff8026179a>] ? __writepage+0xa/0x27
  [<ffffffff80261f57>] ? write_cache_pages+0x1bb/0x30d
  [<ffffffff80261790>] ? __writepage+0x0/0x27
  [<ffffffff802620ec>] ? do_writepages+0x27/0x2d
  [<ffffffff8029f66e>] ? __writeback_single_inode+0x16f/0x35e
  [<ffffffff8029fc8b>] ? sync_sb_inodes+0x283/0x3b8
  [<ffffffff802a0014>] ? writeback_inodes+0x89/0xde
  [<ffffffff80262620>] ? background_writeout+0x8f/0xc9
  [<ffffffff80262c74>] ? pdflush+0x122/0x1c9
  [<ffffffff80262591>] ? background_writeout+0x0/0xc9
  [<ffffffff80262b52>] ? pdflush+0x0/0x1c9
  [<ffffffff8023e3e7>] ? kthread+0x47/0x73
  [<ffffffff8022ae3a>] ? schedule_tail+0x27/0x5c
  [<ffffffff8020bf78>] ? child_rip+0xa/0x12
  [<ffffffff8023e3a0>] ? kthread+0x0/0x73
  [<ffffffff8020bf6e>] ? child_rip+0x0/0x12


Code: 3e 8c fc ff 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 47 18 48 8b 00 48 8b 50 68 48 8b 07 a8 01 75 04 0f 0b eb fe f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c 
RIP  [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
  RSP <ffff81007f329c20>
---[ end trace 82357a26b61bf3d2 ]---
------------[ cut here ]------------
kernel BUG at fs/ext4/inode.c:1931!
invalid opcode: 0000 [2] SMP 
CPU 3 
Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc sg floppy ohci_hcd i2c_amd756 k8temp i2c_core button usbcore
Pid: 7896, comm: dir_check Tainted: G      D   2.6.26-rc5 #4
RIP: 0010:[<ffffffff802dd1c5>]  [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
RSP: 0018:ffff810172ee1d78  EFLAGS: 00010246
RAX: 0a0000000001002d RBX: ffffe200063cd890 RCX: 00000000ffffffe8
RDX: 0000000000004278 RSI: ffff810172ee1eb8 RDI: ffffe200063cd890
RBP: ffff810172ee1eb8 R08: ffff81027ff054c0 R09: ffff810172ee1cb8
R10: ffff810250e6ac80 R11: ffff810250e6a080 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
FS:  00007f3778d0e6f0(0000) GS:ffff81027ff0e000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f3778d25000 CR3: 000000027ecd8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dir_check (pid: 7896, threadinfo ffff810172ee0000, task ffff81017f0fa6f0)
Stack:  0000000000000000 ffff81024ed0c8c0 ffff810172ee1eb8 ffffffff8026179a
  ffffe200063cd890 ffffffff80261f57 ffff81024ed0c8c0 ffffffff80261790
  ffff81024ed0c8c0 0000000500000000 ffff8101ff6caea0 0007ffffffffffff
Call Trace:
  [<ffffffff8026179a>] ? __writepage+0xa/0x27
  [<ffffffff80261f57>] ? write_cache_pages+0x1bb/0x30d
  [<ffffffff80261790>] ? __writepage+0x0/0x27
  [<ffffffff8028ea23>] ? fasync_helper+0x47/0xc8
  [<ffffffff80286f15>] ? cp_new_stat+0xe9/0xfc
  [<ffffffff802921b3>] ? __posix_lock_file+0x43d/0x44e
  [<ffffffff802620ec>] ? do_writepages+0x27/0x2d
  [<ffffffff8025c44f>] ? __filemap_fdatawrite_range+0x4a/0x55
  [<ffffffff802568c8>] ? audit_syscall_exit+0x2ee/0x30c
  [<ffffffff802a28e1>] ? do_fsync+0x2f/0x87
  [<ffffffff80270af7>] ? sys_msync+0x107/0x178
  [<ffffffff8020b2d2>] ? tracesys+0xd5/0xda


Code: 3e 8c fc ff 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 47 18 48 8b 00 48 8b 50 68 48 8b 07 a8 01 75 04 0f 0b eb fe f6 c4 08 75 04 <0f> 0b eb fe 48 89 d0 81 e2 ff 0f 00 00 48 8b 77 10 48 c1 f8 0c 
RIP  [<ffffffff802dd1c5>] ext4_normal_writepage+0x27/0xd8
  RSP <ffff810172ee1d78>
---[ end trace 82357a26b61bf3d2 ]---
SysRq : Show Blocked State
   task                        PC stack   pid father
dir_check     D ffff810181e03eb8     0  7873      1
  ffff8101eea3dc48 0000000000000082 0000000000000096 ffff81007ec39ec0
  ffffffff80600cc0 0000000000000000 ffff81027ff8a658 ffff8101eea21bd0
  ffff81007f395370 ffff8101eea21e08 000000017ff8a640 ffff8101eea21e08
Call Trace:
  [<ffffffff80224a67>] __wake_up+0x38/0x4f
  [<ffffffff8025b87d>] sync_page+0x0/0x41
  [<ffffffff804660f0>] io_schedule+0x2d/0x39
  [<ffffffff8025b8b9>] sync_page+0x3c/0x41
  [<ffffffff804663d4>] __wait_on_bit+0x41/0x70
  [<ffffffff8025bb32>] wait_on_page_bit+0x6b/0x71
  [<ffffffff8023e724>] wake_bit_function+0x0/0x23
  [<ffffffff8026940b>] do_wp_page+0x46a/0x4ba
  [<ffffffff8026aac1>] handle_mm_fault+0x62d/0x6a3
  [<ffffffff80299236>] mntput_no_expire+0x27/0x11e
  [<ffffffff8021ed0c>] do_page_fault+0x435/0x825
  [<ffffffff802921b3>] __posix_lock_file+0x43d/0x44e
  [<ffffffff8029260e>] fcntl_setlk+0x285/0x296
  [<ffffffff802568c8>] audit_syscall_exit+0x2ee/0x30c
  [<ffffffff80467699>] error_exit+0x0/0x51

Holger

Re: Performance of ext4

[Theodore Tso]

On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>
> Doing some performance test between ext3 and ext4 I noticed that ext4
> is not much faster or in some cases slower then ext3. Two years ago when
> I tested ext4 it was a lot faster then ext3 (see my mail:
> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
> I got the following results:

Hi Holger,

You didn't say exactly which version of the kernel/ext4 you were
testing, but a recent change which we made to ext4, but which hasn't
been made to ext3 yet is that barrier support has been enabled to
improve filesystem safety; unfortunately this does imply with it a
slight performance slowdown, which would be more pronounced on
benchmarks with small filesystems.  So when you mount the filesystem
for ext3 and ext4 for benchmarking purposes, you should consistently
use a mount options of barrier=1 or barrier=0.  With ext4, you can use
the mount option "barrier=1,journal_async_commit" which should
ameliorate part of the performance decrease.  The reason why it is not
yet the default is it requires support from e2fsprogs that has not
been released except in development git branches; but as long as you
don't require running e2fsck on uncleanly shutdown systems (probably
not necessary if you are just benchmarking), you can use
journal_async_commit in good health.

Another change which might help out the bonnie benchmark, but which
again requires the latest version of e2fsprogs is to create the
filesystem with flex_bg filesystem feature.  In fact, for
convenience's sake, if you are using the latest development version of
e2fsprogs, you can just use the command "mke2fs -t ext4dev /dev/hdXX"
and it will set up the filesystem with the correct filesystem features
for ext4.  (The "ext4dev" sets the test_fs feature, and is basically
there so it's clear we are still trying to finish up ext4 support.)

> 2 years ago I used 2.6.16.8 but the hardware is still the same. So what has
> happened with the performance of ext4? I noticed that 2 years ago I could
> use extents+mballoc+delalloc, now there is only extents+mballoc in the
> current kernels. Could delalloc make the big difference? I saw that
> in Andrew Morton mm tree delalloc is included. Unfortunately when I tried 
> using
> 2.6.26-rc2-mm1 a sync would never return and there where lot of other
> odd things, so I could not do any tests with delalloc.

As Aneesh has mentioned, there were some bugs in version of ext4's
delalloc caused by insufficient testing of the ext4 patch queue some
changes to our locking strategy went in.  That's been fixed in the
latest patch queue, and we're in the process of cleaning up delalloc
before merging it into the mainline kernel.  (When you tested ext4 two
years ago, none of this was yet in mainline, so it's not a matter of
things delalloc "disappearing", but rather that we've been slow
getting to the point where it was ready for merging.

	       	     	      	  - Ted

Re: Performance of ext4

[Holger Kiehl]

On Wed, 11 Jun 2008, Theodore Tso wrote:

> On Wed, Jun 11, 2008 at 08:02:32AM +0000, Holger Kiehl wrote:
>>
>> Doing some performance test between ext3 and ext4 I noticed that ext4
>> is not much faster or in some cases slower then ext3. Two years ago when
>> I tested ext4 it was a lot faster then ext3 (see my mail:
>> http://lkml.org/lkml/2006/6/6/65). Doing some simple tests with bonnie++
>> I got the following results:
>
> Hi Holger,
>
> You didn't say exactly which version of the kernel/ext4 you were
> testing, but a recent change which we made to ext4, but which hasn't
> been made to ext3 yet is that barrier support has been enabled to
> improve filesystem safety; unfortunately this does imply with it a
> slight performance slowdown, which would be more pronounced on
> benchmarks with small filesystems.
>
I think you mean 'small files'? The test where done with a plain 2.6.25.4,
does that already have barriers enabled? While I did some test I hit this bug:

Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 62053 blocks 62053 reqs (0 success)
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 62053 extents scanned, 14086 goal hits, 47967 2^N hits, 0 breaks, 0 lost
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 561 generated and it took 4080175
Jun 11 19:27:39 helena kernel: EXT4-fs: mballoc: 20889593 preallocated, 2761463 discarded
Jun 11 19:28:05 helena kernel: kjournald2 starting.  Commit interval 15 seconds
Jun 11 19:28:05 helena kernel: EXT4 FS on md7, internal journal
Jun 11 19:28:05 helena kernel: EXT4-fs: mounted filesystem with ordered data mode.
Jun 11 19:28:05 helena kernel: EXT4-fs: file extents enabled
Jun 11 19:28:05 helena kernel: EXT4-fs: mballoc enabled
Jun 11 19:28:50 helena kernel: JBD: barrier-based sync failed on md7 - disabling barriers
Jun 11 19:28:50 helena kernel: ------------[ cut here ]------------
Jun 11 19:28:50 helena kernel: kernel BUG at fs/buffer.c:2879!
Jun 11 19:28:50 helena kernel: invalid opcode: 0000 [1] SMP
Jun 11 19:28:50 helena kernel: CPU 0
Jun 11 19:28:50 helena kernel: Modules linked in: w83627hf lm85 hwmon_vid bonding ipt_REJECT xt_tcpudp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables binfmt_misc floppy i2c_amd756 k8temp i2c_core sg ohci_hcd button usbcore
Jun 11 19:28:50 helena kernel: Pid: 26731, comm: kjournald2 Not tainted 2.6.25.4 #12
Jun 11 19:28:50 helena kernel: RIP: 0010:[submit_bh+16/253]  [submit_bh+16/253] submit_bh+0x10/0xfd
Jun 11 19:28:50 helena kernel: RSP: 0018:ffff8101775fdce0  EFLAGS: 00010246
Jun 11 19:28:50 helena kernel: RAX: 000000000000a02b RBX: ffff81005a182d90 RCX: ffff81027d7c7e60
Jun 11 19:28:50 helena kernel: RDX: 0000000100000000 RSI: ffff81005a182d90 RDI: 0000000000000001
Jun 11 19:28:50 helena kernel: RBP: 0000000000000001 R08: ffffffff8054cdd0 R09: 00000000fffff7e8
Jun 11 19:28:50 helena kernel: R10: ffff810001009940 R11: 0000000000000046 R12: ffff8101972ee000
Jun 11 19:28:50 helena kernel: R13: ffff81007ee79d00 R14: ffff8101775fde48 R15: 00000000fffffffb
Jun 11 19:28:50 helena kernel: FS:  00007f31d2ed67a0(0000) GS:ffffffff8057c000(0000) knlGS:0000000000000000
Jun 11 19:28:50 helena kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jun 11 19:28:50 helena kernel: CR2: 00007f13a584a000 CR3: 000000027ee35000 CR4: 00000000000006e0
Jun 11 19:28:50 helena kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 11 19:28:50 helena kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jun 11 19:28:50 helena kernel: Process kjournald2 (pid: 26731, threadinfo ffff8101775fc000, task ffff81013773d680)
Jun 11 19:28:50 helena kernel: Stack:  ffff8101972ee024 ffff81005a182d90 ffff8101972ee000 ffffffff802f5f96
Jun 11 19:28:50 helena kernel:  ffff81000037646d ffffffff8023c8cf 0000000000000003 0000000000000287
Jun 11 19:28:50 helena kernel:  ffff81005a182850 ffff810075df7240 0000000000000000 ffff8101972ee000
Jun 11 19:28:50 helena kernel: Call Trace:
Jun 11 19:28:50 helena kernel:  [journal_submit_commit_record+315/331] ? journal_submit_commit_record+0x13b/0x14b
Jun 11 19:28:50 helena kernel:  [bit_waitqueue+16/163] ? bit_waitqueue+0x10/0xa3
Jun 11 19:28:50 helena kernel:  [jbd2_journal_commit_transaction+2811/3994] ? jbd2_journal_commit_transaction+0xafb/0xf9a
Jun 11 19:28:50 helena kernel:  [lock_timer_base+38/76] ? lock_timer_base+0x26/0x4c
Jun 11 19:28:50 helena kernel:  [kjournald2+223/514] ? kjournald2+0xdf/0x202
Jun 11 19:28:50 helena kernel:  [autoremove_wake_function+0/46] ? autoremove_wake_function+0x0/0x2e
Jun 11 19:28:50 helena kernel:  [kjournald2+0/514] ? kjournald2+0x0/0x202
Jun 11 19:28:50 helena kernel:  [kthread+71/115] ? kthread+0x47/0x73
Jun 11 19:28:50 helena kernel:  [child_rip+10/18] ? child_rip+0xa/0x12
Jun 11 19:28:50 helena kernel:  [kthread+0/115] ? kthread+0x0/0x73
Jun 11 19:28:50 helena kernel:  [child_rip+0/18] ? child_rip+0x0/0x12
Jun 11 19:28:50 helena kernel:
Jun 11 19:28:50 helena kernel:
Jun 11 19:28:50 helena kernel: Code: 5f 08 e8 14 fb ff ff 48 3b 5c 24 08 48 89 df eb eb 41 5b 5b 5b 89 e8 5d 41 5c c3 41 54 55 89 fd 53 48 8b 06 48 89 f3 a8 04 75 04 <0f> 0b eb fe a8 20 75 04 0f 0b eb fe 48 83 7e 38 00 75 04 0f 0b
Jun 11 19:28:50 helena kernel: RIP  [submit_bh+16/253] submit_bh+0x10/0xfd
Jun 11 19:28:50 helena kernel:  RSP <ffff8101775fdce0>
Jun 11 19:28:50 helena kernel: ---[ end trace 5b73613cd770052b ]---

That happened when the filesystem (mounted with barrier=0) was unmounted
and then mounted with barrier=1.

> So when you mount the filesystem
> for ext3 and ext4 for benchmarking purposes, you should consistently
> use a mount options of barrier=1 or barrier=0.  With ext4, you can use
> the mount option "barrier=1,journal_async_commit" which should
> ameliorate part of the performance decrease.  The reason why it is not
> yet the default is it requires support from e2fsprogs that has not
> been released except in development git branches; but as long as you
> don't require running e2fsck on uncleanly shutdown systems (probably
> not necessary if you are just benchmarking), you can use
> journal_async_commit in good health.
>
> Another change which might help out the bonnie benchmark, but which
> again requires the latest version of e2fsprogs is to create the
> filesystem with flex_bg filesystem feature.  In fact, for
> convenience's sake, if you are using the latest development version of
> e2fsprogs, you can just use the command "mke2fs -t ext4dev /dev/hdXX"
> and it will set up the filesystem with the correct filesystem features
> for ext4.  (The "ext4dev" sets the test_fs feature, and is basically
> there so it's clear we are still trying to finish up ext4 support.)
>
Thank you for the detailed suggestions. I will try those and post the
results as soon as I have them.

Holger

Re: Performance of ext4

[Theodore Tso]

On Wed, Jun 11, 2008 at 08:21:50PM +0000, Holger Kiehl wrote:
> I think you mean 'small files'? 

Yes, sorry, I meant "small files".

> The test where done with a plain 2.6.25.4,
> does that already have barriers enabled? While I did some test I hit this 
> bug:

No, 2.6.25.4 does not have barriers enabled.  It's in the latest
post-2.6.25.5 git repository (which also has the fix so that
barriers=1 doesn't cause an OOPS if you are doing the test on a LVM
device which doesn't support barriers).  Sorry, I wasnt sure if you
were using any of the ext4 patches, and I didn't know whether you were
testing on a raw device or an LVM device.

							 - Ted