From: Wu Fengguang <fengguang.wu@intel.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: Carlos Alberto Lopez Perez <clopez@igalia.com>,
Dave Chinner <david@fromorbit.com>,
Andrew Morton <akpm@linux-foundation.org>,
Jan Kara <jack@suse.cz>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
"bugzilla-daemon@bugzilla.kernel.org"
<bugzilla-daemon@bugzilla.kernel.org>,
"daaugusto@gmail.com" <daaugusto@gmail.com>,
"kernel-bugzilla@cygnusx-1.org" <kernel-bugzilla@cygnusx-1.org>,
"listposter@gmail.com" <listposter@gmail.com>,
"justincase@yopmail.com" <justincase@yopmail.com>,
Tao Ma <tm@tao.ma>, Jens Axboe <axboe@kernel.dk>,
Vivek Goyal <vgoyal@redhat.com>
Subject: Re: [Bug 18632] "INFO: task" dpkg "blocked for more than 120 seconds.
Date: Fri, 10 Jun 2011 11:21:49 +0800 [thread overview]
Message-ID: <20110610032149.GA7114@localhost> (raw)
In-Reply-To: <20110609132357.GA9928@infradead.org>
On Thu, Jun 09, 2011 at 09:23:57PM +0800, Christoph Hellwig wrote:
> Here is a patch that removes the XFS-internal writing of data in sync
> and instead trusts the VFS code to get it right, and also does the
> ioend_wait under a lock, to protect against new pending I/Os showing up.
>
> It passed a single xfsqa run so far, so at least it won't blow up
> completely.
>
> Can you check if that helps with your livelock test case? (And yes, I
> should go and try it myself. Will do as soon as I'll get a bit more
> time).
My pleasure. It's easier to compare results on the same test bed and
I've made it very convenient to test patches :)
The patch does improve the sync time.
before patch after patch
--------------------------------------------------
sync time: 18 sync time: 18
sync time: 21 sync time: 15
sync time: 14 sync time: 14
sync time: 20 sync time: 16
sync time: 21 sync time: 14
avg 18.800 avg 15.400
stddev 2.638 stddev 1.497
I also did extra test runs and collect sync(1) call traces by doing
"echo t > /proc/sysrq-trigger" randomly. The ones in XFS are:
[ 78.395664] sync D 0000000000000002 5296 3024 3003 0x00000000
[ 78.396162] ffff8800b4229de8 0000000000000046 ffffffff819252fa 0000000000000000
[ 78.396850] ffff8800b4228000 00000000001d1a40 00000000001d1a40 ffff8800aefa22f0
[ 78.397519] 00000000001d1a40 ffff8800b4229fd8 00000000001d1a40 ffff8800b4229fd8
[ 78.398189] Call Trace:
[ 78.398431] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 78.398778] [<ffffffff812e906f>] ? _xfs_log_force+0x1de/0x226
[ 78.399143] [<ffffffff81092be5>] ? local_clock+0x41/0x5a
[ 78.399493] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 78.399875] [<ffffffff812e906f>] ? _xfs_log_force+0x1de/0x226
[ 78.400239] [<ffffffff812e9074>] _xfs_log_force+0x1e3/0x226
[ 78.400599] [<ffffffff81067c10>] ? try_to_wake_up+0x28b/0x28b
[ 78.400964] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 78.401347] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 78.401715] [<ffffffff812e90cf>] xfs_log_force+0x18/0x3a
[ 78.402064] [<ffffffff813053e8>] xfs_quiesce_data+0x3a/0x96
[ 78.402424] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 78.402777] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 78.403138] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 78.403482] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 78.403835] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 78.404198] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 78.404533] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
[ 221.106751] sync D 0000000000000007 5296 3136 3116 0x00000000
[ 221.107229] ffff8800aee7bb40 0000000000000046 ffffffff819252fa 0000000000000000
[ 221.107903] ffff8800aee7a000 00000000001d1a40 00000000001d1a40 ffff88009c6a45e0
[ 221.108574] 00000000001d1a40 ffff8800aee7bfd8 00000000001d1a40 ffff8800aee7bfd8
[ 221.109246] Call Trace:
[ 221.109486] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 221.109831] [<ffffffff81928012>] ? rwsem_down_failed_common+0xc7/0x103
[ 221.110225] [<ffffffff81092be5>] ? local_clock+0x41/0x5a
[ 221.110574] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 221.110956] [<ffffffff81928012>] ? rwsem_down_failed_common+0xc7/0x103
[ 221.111352] [<ffffffff8192801e>] rwsem_down_failed_common+0xd3/0x103
[ 221.111738] [<ffffffff81928075>] rwsem_down_read_failed+0x12/0x14
[ 221.112116] [<ffffffff813c81a4>] call_rwsem_down_read_failed+0x14/0x30
[ 221.112513] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 221.112855] [<ffffffff812de62b>] ? xfs_ilock+0x30/0xba
[ 221.113199] [<ffffffff8109153b>] ? down_read_nested+0x74/0x89
[ 221.113563] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 221.113904] [<ffffffff812de648>] xfs_ilock+0x4d/0xba
[ 221.114243] [<ffffffff81304ee0>] xfs_wait_ioend_cb+0x1b/0x36
[ 221.114604] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 221.114973] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 221.115348] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 221.115722] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 221.116103] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 221.116473] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 221.116828] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 221.117185] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 221.117553] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 221.117936] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 221.118312] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 221.118689] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 221.119058] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 221.119417] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 221.119771] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 221.120134] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 221.120477] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 221.120828] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 221.121188] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 349.972430] sync D 0000000000000007 5296 3312 3292 0x00000000
[ 349.972907] ffff88009c77fc08 0000000000000046 ffffffff819252fa 0000000100000070
[ 349.973575] ffff88009c77e000 00000000001d1a40 00000000001d1a40 ffff8800b56b0000
[ 349.974250] 00000000001d1a40 ffff88009c77ffd8 00000000001d1a40 ffff88009c77ffd8
[ 349.974921] Call Trace:
[ 349.975159] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 349.975504] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 349.975880] [<ffffffff8108d5c5>] ? prepare_to_wait+0x6c/0x79
[ 349.976238] [<ffffffff812fade3>] xfs_ioend_wait+0x87/0x9f
[ 349.976588] [<ffffffff8108d332>] ? wake_up_bit+0x2a/0x2a
[ 349.976932] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 349.977272] [<ffffffff81304ee8>] xfs_wait_ioend_cb+0x23/0x36
[ 349.977631] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 349.977997] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 349.978371] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 349.978748] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 349.979129] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 349.979499] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 349.979856] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 349.980211] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 349.980582] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 349.980964] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 349.981338] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 349.981717] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 349.982085] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 349.982443] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 349.982798] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 349.983159] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 349.983504] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 349.983857] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 349.984215] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 349.984552] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
[ 419.417496] sync D 0000000000000006 5296 3412 3392 0x00000000
[ 419.417972] ffff8800aef6dc08 0000000000000046 ffffffff819252fa 0000000100000070
[ 419.418645] ffff8800aef6c000 00000000001d1a40 00000000001d1a40 ffff8800b704a2f0
[ 419.419317] 00000000001d1a40 ffff8800aef6dfd8 00000000001d1a40 ffff8800aef6dfd8
[ 419.419987] Call Trace:
[ 419.420228] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 419.420576] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 419.420957] [<ffffffff8108d5c5>] ? prepare_to_wait+0x6c/0x79
[ 419.421321] [<ffffffff812fade3>] xfs_ioend_wait+0x87/0x9f
[ 419.421673] [<ffffffff8108d332>] ? wake_up_bit+0x2a/0x2a
[ 419.422020] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 419.422364] [<ffffffff81304ee8>] xfs_wait_ioend_cb+0x23/0x36
[ 419.422724] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 419.423093] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 419.423470] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 419.423844] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 419.424226] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 419.424596] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 419.424951] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 419.425307] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 419.425676] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 419.426058] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 419.426435] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 419.426812] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 419.427180] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 419.427542] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 419.427895] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 419.428258] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 419.428601] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 419.428954] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 419.429315] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 419.429648] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
Thanks,
Fengguang
next prev parent reply other threads:[~2011-06-10 3:21 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <bug-18632-27@https.bugzilla.kernel.org/>
[not found] ` <201106082138.p58Lchgj002615@demeter2.kernel.org>
[not found] ` <20110608150241.8412a63d.akpm@linux-foundation.org>
2011-06-09 3:32 ` [Bug 18632] "INFO: task" dpkg "blocked for more than 120 seconds Wu Fengguang
2011-06-09 3:54 ` Wu Fengguang
2011-06-09 8:27 ` Christoph Hellwig
2011-06-09 9:09 ` Wu Fengguang
2011-06-09 11:02 ` Christoph Hellwig
2011-06-09 12:11 ` Wu Fengguang
2011-06-09 12:17 ` Wu Fengguang
2011-06-09 12:17 ` Christoph Hellwig
2011-06-09 12:43 ` Wu Fengguang
2011-06-09 13:23 ` Christoph Hellwig
2011-06-10 3:21 ` Wu Fengguang [this message]
2011-06-19 15:56 ` Christoph Hellwig
2011-06-19 16:33 ` Wu Fengguang
2011-06-09 13:56 ` Tao Ma
2011-06-09 14:12 ` Tao Ma
2011-06-09 14:21 ` Christoph Hellwig
2011-06-09 14:32 ` Vivek Goyal
2011-06-09 14:51 ` Tao Ma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110610032149.GA7114@localhost \
--to=fengguang.wu@intel.com \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=bugzilla-daemon@bugzilla.kernel.org \
--cc=clopez@igalia.com \
--cc=daaugusto@gmail.com \
--cc=david@fromorbit.com \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=justincase@yopmail.com \
--cc=kernel-bugzilla@cygnusx-1.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=listposter@gmail.com \
--cc=tm@tao.ma \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.