* Possible deadlock when deleting from realtime section
@ 2010-10-25 15:59 Denny Priebe
2010-10-26 9:19 ` Denny Priebe
2010-10-26 22:58 ` Dave Chinner
0 siblings, 2 replies; 4+ messages in thread
From: Denny Priebe @ 2010-10-25 15:59 UTC (permalink / raw)
To: xfs
Hi,
I'm experiencing a deadlock situation when deleting directories placed in
the realtime section. This is reproducable with kernel versions 2.6.35.7 and
2.6.36-rc8. I haven't tried final 2.6.36 yet. The same setup is working
perfectly without using the realtime section. The file system has been
created with
mkfs.xfs -f -l logdev=/dev/sdb1 -r rtdev=/dev/sdb3,extsize=256k /dev/sdb2
and mounted with
mount -t xfs -o logdev=/dev/sdb1,rtdev=/dev/sdb3,sunit=512,swidth=2048 \
/dev/sdb2 /there
This is where rm blocks:
SysRq : Show Blocked State
task PC stack pid father
rm D 0000000000000005 0 1705 1658 0x00000080
ffff88018c40d858 0000000000000086 ffff880100000000 ffff88018c40c010
ffff88018c40dfd8 00000000000148c0 ffff88018b1c2e20 ffff88018b1c31d8
ffff88018b1c31d0 00000000000148c0 00000000000148c0 ffff88018c40dfd8
Call Trace:
[<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
[<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
[<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff81436e3d>] ? down_write+0x40/0x44
[<ffffffffa033be9b>] xfs_ilock+0x4a/0x9a [xfs]
[<ffffffffa033c413>] xfs_iget+0x34c/0x5cd [xfs]
[<ffffffffa0351fa9>] xfs_trans_iget+0x1b/0x56 [xfs]
[<ffffffffa0313035>] xfs_rtfree_extent+0x37/0xdc [xfs]
[<ffffffffa033fce2>] ? xfs_iext_remove+0xc0/0xd2 [xfs]
[<ffffffffa0324ffe>] ? xfs_bmap_del_extent+0x2e8/0x93d [xfs]
[<ffffffffa0324e87>] xfs_bmap_del_extent+0x171/0x93d [xfs]
[<ffffffffa03506eb>] ? xfs_trans_commit_iclog+0x2ba/0x2d3 [xfs]
[<ffffffffa0325dc3>] xfs_bunmapi+0x770/0xa28 [xfs]
[<ffffffffa033e44e>] xfs_itruncate_finish+0x185/0x2b8 [xfs]
[<ffffffffa0354437>] xfs_inactive+0x1c8/0x3d5 [xfs]
[<ffffffffa035f38c>] xfs_fs_evict_inode+0xd5/0xdd [xfs]
[<ffffffff8111282b>] evict+0x22/0x92
[<ffffffff81112c60>] iput+0x1bc/0x225
[<ffffffff8110b079>] do_unlinkat+0x103/0x156
[<ffffffff811086c5>] ? path_put+0x1d/0x22
[<ffffffff81093e2c>] ? audit_syscall_entry+0x119/0x145
[<ffffffff8110b205>] sys_unlinkat+0x24/0x26
[<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b
Thanks in advance for any feedback,
Denny
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Possible deadlock when deleting from realtime section
2010-10-25 15:59 Possible deadlock when deleting from realtime section Denny Priebe
@ 2010-10-26 9:19 ` Denny Priebe
2010-10-26 22:58 ` Dave Chinner
1 sibling, 0 replies; 4+ messages in thread
From: Denny Priebe @ 2010-10-26 9:19 UTC (permalink / raw)
To: xfs
After that, recovery hangs as well:
SGI XFS with ACLs, security attributes, realtime, large block/inode
numbers, debug enabled
SGI XFS Quota Management subsystem
Filesystem "sdb2": Disabling barriers, not supported with external log device
XFS mounting filesystem sdb2
Starting XFS recovery on filesystem: sdb2 (logdev: /dev/sdb1)
SysRq : Show Blocked State
task PC stack pid father
mount D 0000000000000000 0 1481 1464 0x00000080
ffff880186c17558 0000000000000086 ffff880100000000 ffff880186c16010
ffff880186c17fd8 00000000000148c0 ffff880187364530 ffff8801873648e8
ffff8801873648e0 00000000000148c0 00000000000148c0 ffff880186c17fd8
Call Trace:
[<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
[<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
[<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff81436e3d>] ? down_write+0x40/0x44
[<ffffffffa032b7d0>] xfs_ilock+0xb7/0x114 [xfs]
[<ffffffffa032bead>] xfs_iget+0x34c/0x61b [xfs]
[<ffffffffa03479b7>] xfs_trans_iget+0x1b/0x54 [xfs]
[<ffffffffa02f588e>] xfs_rtfree_extent+0x37/0x117 [xfs]
[<ffffffffa03311f2>] ? xfs_iext_remove+0xdd/0xef [xfs]
[<ffffffffa030b52f>] ? xfs_bmap_del_extent+0x40c/0xab5 [xfs]
[<ffffffffa030b3ad>] xfs_bmap_del_extent+0x28a/0xab5 [xfs]
[<ffffffffa030dce8>] xfs_bunmapi+0xa0c/0xdd2 [xfs]
[<ffffffff810f17ef>] ? new_slab+0x199/0x1c0
[<ffffffffa032e26a>] xfs_itruncate_finish+0x32e/0x4f1 [xfs]
[<ffffffffa034a5d1>] xfs_inactive+0x224/0x47a [xfs]
[<ffffffffa035615d>] xfs_fs_evict_inode+0x101/0x109 [xfs]
[<ffffffff8111282b>] evict+0x22/0x92
[<ffffffff81112c60>] iput+0x1bc/0x225
[<ffffffffa033b84a>] xlog_recover_process_one_iunlink+0x118/0x17e [xfs]
[<ffffffffa033b930>] xlog_recover_process_iunlinks+0x80/0xf7 [xfs]
[<ffffffffa033bc5b>] xlog_recover_finish+0x54/0xbc [xfs]
[<ffffffffa0339e46>] xfs_log_mount_finish+0x1c/0x44 [xfs]
[<ffffffffa0342832>] xfs_mountfs+0x559/0x683 [xfs]
[<ffffffffa034c569>] ? kmem_zalloc+0x11/0x2c [xfs]
[<ffffffffa0342bfb>] ? xfs_mru_cache_create+0x117/0x146 [xfs]
[<ffffffffa0357a03>] xfs_fs_fill_super+0x1d9/0x316 [xfs]
[<ffffffff81102c48>] get_sb_bdev+0x14b/0x1ae
[<ffffffffa035782a>] ? xfs_fs_fill_super+0x0/0x316 [xfs]
[<ffffffffa0355c54>] xfs_fs_get_sb+0x13/0x15 [xfs]
[<ffffffff8110229f>] vfs_kern_mount+0xb8/0x1a2
[<ffffffff811023e7>] do_kern_mount+0x48/0xe8
[<ffffffff811175ef>] do_mount+0x74f/0x7c6
[<ffffffff811176e9>] sys_mount+0x83/0xbd
[<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b
Regards,
Denny
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Possible deadlock when deleting from realtime section
2010-10-25 15:59 Possible deadlock when deleting from realtime section Denny Priebe
2010-10-26 9:19 ` Denny Priebe
@ 2010-10-26 22:58 ` Dave Chinner
2010-12-02 12:20 ` Denny Priebe
1 sibling, 1 reply; 4+ messages in thread
From: Dave Chinner @ 2010-10-26 22:58 UTC (permalink / raw)
To: Denny Priebe; +Cc: xfs
On Mon, Oct 25, 2010 at 03:59:22PM +0000, Denny Priebe wrote:
> Hi,
>
> I'm experiencing a deadlock situation when deleting directories placed in
> the realtime section. This is reproducable with kernel versions 2.6.35.7 and
> 2.6.36-rc8. I haven't tried final 2.6.36 yet. The same setup is working
> perfectly without using the realtime section. The file system has been
> created with
>
> mkfs.xfs -f -l logdev=/dev/sdb1 -r rtdev=/dev/sdb3,extsize=256k /dev/sdb2
FYI, using an external log on the same device as the main device will be
slower than using the default internal log....
> and mounted with
>
> mount -t xfs -o logdev=/dev/sdb1,rtdev=/dev/sdb3,sunit=512,swidth=2048 \
> /dev/sdb2 /there
>
> This is where rm blocks:
>
> SysRq : Show Blocked State
> task PC stack pid father
> rm D 0000000000000005 0 1705 1658 0x00000080
> ffff88018c40d858 0000000000000086 ffff880100000000 ffff88018c40c010
> ffff88018c40dfd8 00000000000148c0 ffff88018b1c2e20 ffff88018b1c31d8
> ffff88018b1c31d0 00000000000148c0 00000000000148c0 ffff88018c40dfd8
> Call Trace:
> [<ffffffff81437739>] rwsem_down_failed_common+0xd3/0x105
> [<ffffffff8143777e>] rwsem_down_write_failed+0x13/0x15
> [<ffffffff81202ad3>] call_rwsem_down_write_failed+0x13/0x20
> [<ffffffff81436e3d>] ? down_write+0x40/0x44
> [<ffffffffa033be9b>] xfs_ilock+0x4a/0x9a [xfs]
> [<ffffffffa033c413>] xfs_iget+0x34c/0x5cd [xfs]
> [<ffffffffa0351fa9>] xfs_trans_iget+0x1b/0x56 [xfs]
> [<ffffffffa0313035>] xfs_rtfree_extent+0x37/0xdc [xfs]
> [<ffffffffa033fce2>] ? xfs_iext_remove+0xc0/0xd2 [xfs]
> [<ffffffffa0324ffe>] ? xfs_bmap_del_extent+0x2e8/0x93d [xfs]
> [<ffffffffa0324e87>] xfs_bmap_del_extent+0x171/0x93d [xfs]
> [<ffffffffa03506eb>] ? xfs_trans_commit_iclog+0x2ba/0x2d3 [xfs]
> [<ffffffffa0325dc3>] xfs_bunmapi+0x770/0xa28 [xfs]
> [<ffffffffa033e44e>] xfs_itruncate_finish+0x185/0x2b8 [xfs]
> [<ffffffffa0354437>] xfs_inactive+0x1c8/0x3d5 [xfs]
> [<ffffffffa035f38c>] xfs_fs_evict_inode+0xd5/0xdd [xfs]
> [<ffffffff8111282b>] evict+0x22/0x92
> [<ffffffff81112c60>] iput+0x1bc/0x225
> [<ffffffff8110b079>] do_unlinkat+0x103/0x156
> [<ffffffff811086c5>] ? path_put+0x1d/0x22
> [<ffffffff81093e2c>] ? audit_syscall_entry+0x119/0x145
> [<ffffffff8110b205>] sys_unlinkat+0x24/0x26
> [<ffffffff81009ac2>] system_call_fastpath+0x16/0x1b
So it is blocked trying to lock the allocation bitmap inode.
Hmmm, I suspect that XFS_ITRUNC_MAX_EXTENTS is the start of
the problem here.
i.e. what I think might be the problem is that xfs_bunmapi() is
trying to free two extents in the one transaction, and what we see
above is the second extent being freed via xfs_rtfree_extent().
The bitmap inode won't be unlocked until the transaction commits,
so the second call to xfs_trans_iget() in the same transaction will
hang like this.
Hmmm. Looks like we broke recursive inode locking in
xfs_trans_iget() in commit aa72a5cf00001d0b952c7c755be404b9118ceb2e
("xfs: simplify xfs_trans_iget"). The changelog says:
"....
A quick audit of the callers of xfs_trans_iget shows that no
caller really relies on this behaviour fortunately - xfs_ialloc
allows this inode from disk so it must not be there before, and
all the RT allocator routines only every add each RT bitmap
inode once.
...."
What we didn't take into account is multiple RT allocator calls in
the same transaction context. Let me think about the best way to fix
this.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: Possible deadlock when deleting from realtime section
2010-10-26 22:58 ` Dave Chinner
@ 2010-12-02 12:20 ` Denny Priebe
0 siblings, 0 replies; 4+ messages in thread
From: Denny Priebe @ 2010-12-02 12:20 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On Tue, Oct 26, 2010 at 10:58 PM, Dave Chinner <david@fromorbit.com> wrote:
> Hmmm. Looks like we broke recursive inode locking in
> xfs_trans_iget() in commit aa72a5cf00001d0b952c7c755be404b9118ceb2e
> ("xfs: simplify xfs_trans_iget").
> What we didn't take into account is multiple RT allocator calls in
> the same transaction context. Let me think about the best way to fix
> this.
Is there any news on this? Maybe some patch I could try out?
Regards,
Denny
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-12-02 12:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-10-25 15:59 Possible deadlock when deleting from realtime section Denny Priebe
2010-10-26 9:19 ` Denny Priebe
2010-10-26 22:58 ` Dave Chinner
2010-12-02 12:20 ` Denny Priebe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox