From: Wu Fengguang <fengguang.wu@intel.com>
To: Christoph Hellwig <hch@infradead.org>
Cc: Carlos Alberto Lopez Perez <clopez@igalia.com>,
Dave Chinner <david@fromorbit.com>,
Andrew Morton <akpm@linux-foundation.org>,
Jan Kara <jack@suse.cz>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
"bugzilla-daemon@bugzilla.kernel.org"
<bugzilla-daemon@bugzilla.kernel.org>,
"daaugusto@gmail.com" <daaugusto@gmail.com>,
"kernel-bugzilla@cygnusx-1.org" <kernel-bugzilla@cygnusx-1.org>,
"listposter@gmail.com" <listposter@gmail.com>,
"justincase@yopmail.com" <justincase@yopmail.com>,
Tao Ma <tm@tao.ma>, Jens Axboe <axboe@kernel.dk>,
Vivek Goyal <vgoyal@redhat.com>
Subject: Re: [Bug 18632] "INFO: task" dpkg "blocked for more than 120 seconds.
Date: Fri, 10 Jun 2011 11:21:49 +0800 [thread overview]
Message-ID: <20110610032149.GA7114@localhost> (raw)
In-Reply-To: <20110609132357.GA9928@infradead.org>
On Thu, Jun 09, 2011 at 09:23:57PM +0800, Christoph Hellwig wrote:
> Here is a patch that removes the XFS-internal writing of data in sync
> and instead trusts the VFS code to get it right, and also does the
> ioend_wait under a lock, to protect against new pending I/Os showing up.
>
> It passed a single xfsqa run so far, so at least it won't blow up
> completely.
>
> Can you check if that helps with your livelock test case? (And yes, I
> should go and try it myself. Will do as soon as I'll get a bit more
> time).
My pleasure. It's easier to compare results on the same test bed and
I've made it very convenient to test patches :)
The patch does improve the sync time.
before patch after patch
--------------------------------------------------
sync time: 18 sync time: 18
sync time: 21 sync time: 15
sync time: 14 sync time: 14
sync time: 20 sync time: 16
sync time: 21 sync time: 14
avg 18.800 avg 15.400
stddev 2.638 stddev 1.497
I also did extra test runs and collect sync(1) call traces by doing
"echo t > /proc/sysrq-trigger" randomly. The ones in XFS are:
[ 78.395664] sync D 0000000000000002 5296 3024 3003 0x00000000
[ 78.396162] ffff8800b4229de8 0000000000000046 ffffffff819252fa 0000000000000000
[ 78.396850] ffff8800b4228000 00000000001d1a40 00000000001d1a40 ffff8800aefa22f0
[ 78.397519] 00000000001d1a40 ffff8800b4229fd8 00000000001d1a40 ffff8800b4229fd8
[ 78.398189] Call Trace:
[ 78.398431] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 78.398778] [<ffffffff812e906f>] ? _xfs_log_force+0x1de/0x226
[ 78.399143] [<ffffffff81092be5>] ? local_clock+0x41/0x5a
[ 78.399493] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 78.399875] [<ffffffff812e906f>] ? _xfs_log_force+0x1de/0x226
[ 78.400239] [<ffffffff812e9074>] _xfs_log_force+0x1e3/0x226
[ 78.400599] [<ffffffff81067c10>] ? try_to_wake_up+0x28b/0x28b
[ 78.400964] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 78.401347] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 78.401715] [<ffffffff812e90cf>] xfs_log_force+0x18/0x3a
[ 78.402064] [<ffffffff813053e8>] xfs_quiesce_data+0x3a/0x96
[ 78.402424] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 78.402777] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 78.403138] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 78.403482] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 78.403835] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 78.404198] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 78.404533] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
[ 221.106751] sync D 0000000000000007 5296 3136 3116 0x00000000
[ 221.107229] ffff8800aee7bb40 0000000000000046 ffffffff819252fa 0000000000000000
[ 221.107903] ffff8800aee7a000 00000000001d1a40 00000000001d1a40 ffff88009c6a45e0
[ 221.108574] 00000000001d1a40 ffff8800aee7bfd8 00000000001d1a40 ffff8800aee7bfd8
[ 221.109246] Call Trace:
[ 221.109486] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 221.109831] [<ffffffff81928012>] ? rwsem_down_failed_common+0xc7/0x103
[ 221.110225] [<ffffffff81092be5>] ? local_clock+0x41/0x5a
[ 221.110574] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 221.110956] [<ffffffff81928012>] ? rwsem_down_failed_common+0xc7/0x103
[ 221.111352] [<ffffffff8192801e>] rwsem_down_failed_common+0xd3/0x103
[ 221.111738] [<ffffffff81928075>] rwsem_down_read_failed+0x12/0x14
[ 221.112116] [<ffffffff813c81a4>] call_rwsem_down_read_failed+0x14/0x30
[ 221.112513] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 221.112855] [<ffffffff812de62b>] ? xfs_ilock+0x30/0xba
[ 221.113199] [<ffffffff8109153b>] ? down_read_nested+0x74/0x89
[ 221.113563] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 221.113904] [<ffffffff812de648>] xfs_ilock+0x4d/0xba
[ 221.114243] [<ffffffff81304ee0>] xfs_wait_ioend_cb+0x1b/0x36
[ 221.114604] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 221.114973] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 221.115348] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 221.115722] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 221.116103] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 221.116473] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 221.116828] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 221.117185] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 221.117553] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 221.117936] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 221.118312] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 221.118689] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 221.119058] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 221.119417] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 221.119771] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 221.120134] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 221.120477] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 221.120828] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 221.121188] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 349.972430] sync D 0000000000000007 5296 3312 3292 0x00000000
[ 349.972907] ffff88009c77fc08 0000000000000046 ffffffff819252fa 0000000100000070
[ 349.973575] ffff88009c77e000 00000000001d1a40 00000000001d1a40 ffff8800b56b0000
[ 349.974250] 00000000001d1a40 ffff88009c77ffd8 00000000001d1a40 ffff88009c77ffd8
[ 349.974921] Call Trace:
[ 349.975159] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 349.975504] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 349.975880] [<ffffffff8108d5c5>] ? prepare_to_wait+0x6c/0x79
[ 349.976238] [<ffffffff812fade3>] xfs_ioend_wait+0x87/0x9f
[ 349.976588] [<ffffffff8108d332>] ? wake_up_bit+0x2a/0x2a
[ 349.976932] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 349.977272] [<ffffffff81304ee8>] xfs_wait_ioend_cb+0x23/0x36
[ 349.977631] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 349.977997] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 349.978371] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 349.978748] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 349.979129] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 349.979499] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 349.979856] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 349.980211] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 349.980582] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 349.980964] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 349.981338] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 349.981717] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 349.982085] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 349.982443] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 349.982798] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 349.983159] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 349.983504] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 349.983857] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 349.984215] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 349.984552] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
[ 419.417496] sync D 0000000000000006 5296 3412 3392 0x00000000
[ 419.417972] ffff8800aef6dc08 0000000000000046 ffffffff819252fa 0000000100000070
[ 419.418645] ffff8800aef6c000 00000000001d1a40 00000000001d1a40 ffff8800b704a2f0
[ 419.419317] 00000000001d1a40 ffff8800aef6dfd8 00000000001d1a40 ffff8800aef6dfd8
[ 419.419987] Call Trace:
[ 419.420228] [<ffffffff819252fa>] ? schedule+0x374/0x992
[ 419.420576] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 419.420957] [<ffffffff8108d5c5>] ? prepare_to_wait+0x6c/0x79
[ 419.421321] [<ffffffff812fade3>] xfs_ioend_wait+0x87/0x9f
[ 419.421673] [<ffffffff8108d332>] ? wake_up_bit+0x2a/0x2a
[ 419.422020] [<ffffffff812de648>] ? xfs_ilock+0x4d/0xba
[ 419.422364] [<ffffffff81304ee8>] xfs_wait_ioend_cb+0x23/0x36
[ 419.422724] [<ffffffff813051a9>] xfs_inode_ag_walk+0x1a5/0x288
[ 419.423093] [<ffffffff81305263>] ? xfs_inode_ag_walk+0x25f/0x288
[ 419.423470] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 419.423844] [<ffffffff8109d329>] ? lock_release_holdtime+0xa3/0xab
[ 419.424226] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 419.424596] [<ffffffff812efbfc>] ? xfs_perag_get+0x68/0xd1
[ 419.424951] [<ffffffff812efc14>] ? xfs_perag_get+0x80/0xd1
[ 419.425307] [<ffffffff812efb94>] ? xfs_check_sizes+0x160/0x160
[ 419.425676] [<ffffffff813052f9>] ? xfs_inode_ag_iterator+0x6d/0x8f
[ 419.426058] [<ffffffff81304ec5>] ? xfs_sync_inode_data+0x9d/0x9d
[ 419.426435] [<ffffffff813052d3>] xfs_inode_ag_iterator+0x47/0x8f
[ 419.426812] [<ffffffff81164771>] ? __sync_filesystem+0x7a/0x7a
[ 419.427180] [<ffffffff813053d1>] xfs_quiesce_data+0x23/0x96
[ 419.427542] [<ffffffff81302f82>] xfs_fs_sync_fs+0x21/0x48
[ 419.427895] [<ffffffff8116475d>] __sync_filesystem+0x66/0x7a
[ 419.428258] [<ffffffff81164787>] sync_one_sb+0x16/0x18
[ 419.428601] [<ffffffff81142d37>] iterate_supers+0x72/0xc8
[ 419.428954] [<ffffffff811646bc>] sync_filesystems+0x20/0x22
[ 419.429315] [<ffffffff811647fa>] sys_sync+0x21/0x33
[ 419.429648] [<ffffffff8192f742>] system_call_fastpath+0x16/0x1b
Thanks,
Fengguang
next prev parent reply other threads:[~2011-06-10 3:21 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <bug-18632-27@https.bugzilla.kernel.org/>
[not found] ` <201106082138.p58Lchgj002615@demeter2.kernel.org>
[not found] ` <20110608150241.8412a63d.akpm@linux-foundation.org>
2011-06-09 3:32 ` [Bug 18632] "INFO: task" dpkg "blocked for more than 120 seconds Wu Fengguang
2011-06-09 3:54 ` Wu Fengguang
2011-06-09 8:27 ` Christoph Hellwig
2011-06-09 9:09 ` Wu Fengguang
2011-06-09 11:02 ` Christoph Hellwig
2011-06-09 12:11 ` Wu Fengguang
2011-06-09 12:17 ` Wu Fengguang
2011-06-09 12:17 ` Christoph Hellwig
2011-06-09 12:43 ` Wu Fengguang
2011-06-09 13:23 ` Christoph Hellwig
2011-06-10 3:21 ` Wu Fengguang [this message]
2011-06-19 15:56 ` Christoph Hellwig
2011-06-19 16:33 ` Wu Fengguang
2011-06-09 13:56 ` Tao Ma
2011-06-09 14:12 ` Tao Ma
2011-06-09 14:21 ` Christoph Hellwig
2011-06-09 14:32 ` Vivek Goyal
2011-06-09 14:51 ` Tao Ma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110610032149.GA7114@localhost \
--to=fengguang.wu@intel.com \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=bugzilla-daemon@bugzilla.kernel.org \
--cc=clopez@igalia.com \
--cc=daaugusto@gmail.com \
--cc=david@fromorbit.com \
--cc=hch@infradead.org \
--cc=jack@suse.cz \
--cc=justincase@yopmail.com \
--cc=kernel-bugzilla@cygnusx-1.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=listposter@gmail.com \
--cc=tm@tao.ma \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).