From: Liu Bo <bo.li.liu@oracle.com>
To: Mitch Harder <mitch.harder@sabayonlinux.org>
Cc: linux-btrfs@vger.kernel.org, chris.mason@fusionio.com,
JBacik@fusionio.com, dave@jikos.cz, kitayama@cl.bb4u.ne.jp,
miaox@cn.fujitsu.com
Subject: Re: [PATCH V5] Btrfs: snapshot-aware defrag
Date: Fri, 25 Jan 2013 23:42:54 +0800 [thread overview]
Message-ID: <20130125154236.GA3997@liubo> (raw)
In-Reply-To: <CAKcLGm_Casj44A3OL3hpsYAL9szb3-F+8C2fsLow9Trk_y=7dg@mail.gmail.com>
On Fri, Jan 25, 2013 at 08:55:58AM -0600, Mitch Harder wrote:
> On Wed, Jan 23, 2013 at 6:52 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> > On Wed, Jan 23, 2013 at 10:05:04AM -0600, Mitch Harder wrote:
> >> On Wed, Jan 23, 2013 at 1:51 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >> > On Tue, Jan 22, 2013 at 11:41:19AM -0600, Mitch Harder wrote:
> >> >> On Thu, Jan 17, 2013 at 8:42 AM, Mitch Harder
> >> >> <mitch.harder@sabayonlinux.org> wrote:
> >> >> > On Wed, Jan 16, 2013 at 6:36 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >> >> >> This comes from one of btrfs's project ideas,
> >> >> >> As we defragment files, we break any sharing from other snapshots.
> >> >> >> The balancing code will preserve the sharing, and defrag needs to grow this
> >> >> >> as well.
> >> > [...]
> >> >> >
> >> >> > I've been testing this patch on a 3.7.2 kernel merged with the
> >> >> > for-linus branch for the 3.8_rc kernels, and I'm seeing the following
> >> >> > error:
> >> >> >
> >> >>
> >> >> I've reproduced the error with CONFIG_DEBUG_LIST enabled, which shows
> >> >> some problem with an entry in the list.
> >> >>
> >> >> [59312.260441] ------------[ cut here ]------------
> >> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> >> >> [59312.260458] Hardware name: OptiPlex 745
> >> >> [59312.260461] list_del corruption. next->prev should be
> >> >> ffff88006511c438, but was dead000000200200
> >> >
> >> > LIST_POISON2 -> (000000200200)
> >> > So we can know that the next one is deleted from the list even _earlier_
> >> > than the current one is.
> >> >
> >> > Any other messages before this warning complains?
> >> >
> >>
> >> Just some normal feedback from a metadata balance I had run.
> >
> > Well, these do fit my expectation, since balance also involves with playing with
> > root_list, which may lead to the bad situation.
> >
> >>
> >> [14057.193343] device fsid 28c688c5-7dbd-4071-b271-1bf6726d8835 devid
> >> 1 transid 4 /dev/sda7
> >> [14057.194438] btrfs: force lzo compression
> >> [14057.194446] btrfs: enabling auto defrag
> >> [14057.194449] btrfs: disk space caching is enabled
> >> [14057.194452] btrfs flagging fs with big metadata feature
> >> [14057.194455] btrfs: lzo incompat flag set.
> >> [57508.799193] btrfs: relocating block group 14516486144 flags 4
> >> [57632.178797] btrfs: found 6775 extents
> >> [57633.214701] btrfs: relocating block group 11832131584 flags 4
> >> [57776.400102] btrfs: found 6480 extents
> >> [57777.021175] btrfs: relocating block group 10489954304 flags 4
> >> [57949.182725] btrfs: found 6681 extents
> >> [59312.260441] ------------[ cut here ]------------
> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> >> [59312.260458] Hardware name: OptiPlex 745
> >> ...
> >>
> >> I'm going to try to wrap some debugging around the section of code in
> >> btrfs_clean_old_snapshots() where the dead_roots list is spliced onto
> >> the root list being processed. The double entry may be slipping in
> >> here.
> >>
> >> 1764 spin_lock(&fs_info->trans_lock);
> >> 1765 list_splice_init(&fs_info->dead_roots, &list);
> >> 1766 spin_unlock(&fs_info->trans_lock);
> >
> > hmm, I don't think there is anything wrong in this code. But you can
> > give it a shot anyway :)
> >
>
> I've changed up my reproducer to try some things that may hit the
> issue quicker and more reliably.
>
> It gave me a slightly different set of warnings in dmesg, which seem
> to suggest issues in the dead_root list.
Great! Many thanks for nail it down, we really shouldn't iput()
after btrfs_iget().
Could you please try this(remove iput()) and see if it gets us rid of
the trouble?
thanks,
liubo
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 1683f48..c7a0fb7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -2337,7 +2337,6 @@ out_free_path:
out_unlock:
unlock_extent_cached(&BTRFS_I(inode)->io_tree, lock_start,
lock_end,
&cached, GFP_NOFS);
- iput(inode);
return ret;
}
>
> [43925.656065] device fsid a8f6fadb-3022-4c01-b369-f1f3f638c052 devid
> 1 transid 310 /dev/sda7
> [43925.658062] btrfs: force lzo compression
> [43925.658072] btrfs: enabling auto defrag
> [43925.658075] btrfs: disk space caching is enabled
> [43925.658078] btrfs: lzo incompat flag set.
> [44503.421293] btrfs: unlinked 1 orphans
> [44898.287365] btrfs: unlinked 1 orphans
> [45080.641383] btrfs: unlinked 1 orphans
> [45250.063773] btrfs: unlinked 1 orphans
> [46223.387355] btrfs: unlinked 1 orphans
> [46476.473944] btrfs: unlinked 1 orphans
> [46499.665615] btrfs: unlinked 1 orphans
> [46769.785454] ------------[ cut here ]------------
> [46769.785471] WARNING: at lib/list_debug.c:36 __list_add+0x9d/0xba()
> [46769.785474] Hardware name: OptiPlex 745
> [46769.785478] list_add double add: new=ffff880050c27c38,
> prev=ffff880078f3e720, next=ffff880050c27c38.
> [46769.785480] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46769.785537] Pid: 18291, comm: btrfs-endio-wri Not tainted 3.7.4-sad-v1+ #3
> [46769.785539] Call Trace:
> [46769.785549] [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46769.785553] [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46769.785558] [<ffffffff8120987b>] __list_add+0x9d/0xba
> [46769.785586] [<ffffffffa0179dd6>] btrfs_add_dead_root+0x42/0x56 [btrfs]
> [46769.785603] [<ffffffffa0187b67>] btrfs_destroy_inode+0x227/0x25b [btrfs]
> [46769.785611] [<ffffffff8111393a>] destroy_inode+0x3b/0x54
> [46769.785615] [<ffffffff81113a9c>] evict+0x149/0x151
> [46769.785619] [<ffffffff81114322>] iput+0x12c/0x135
> [46769.785636] [<ffffffffa018455f>] relink_extent_backref+0x669/0x6af [btrfs]
> [46769.785642] [<ffffffff815e9849>] ? __slab_free+0x17c/0x21b
> [46769.785658] [<ffffffffa0184d15>] ?
> btrfs_finish_ordered_io+0x770/0x827 [btrfs]
> [46769.785674] [<ffffffffa0184ce5>] btrfs_finish_ordered_io+0x740/0x827 [btrfs]
> [46769.785691] [<ffffffffa0184de1>] finish_ordered_fn+0x15/0x17 [btrfs]
> [46769.785706] [<ffffffffa019e5c9>] worker_loop+0x14c/0x493 [btrfs]
> [46769.785722] [<ffffffffa019e47d>] ? btrfs_queue_worker+0x258/0x258 [btrfs]
> [46769.785728] [<ffffffff8104c750>] kthread+0xba/0xc2
> [46769.785732] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46769.785737] [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46769.785741] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46769.785745] ---[ end trace 7528086f91b151b5 ]---
> [46799.053062] ------------[ cut here ]------------
> [46799.053078] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> [46799.053082] Hardware name: OptiPlex 745
> [46799.053087] list_del corruption. next->prev should be
> ffff880050c27c38, but was ffff8800057fde38
> [46799.053090] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46799.053163] Pid: 18210, comm: btrfs-cleaner Tainted: G W
> 3.7.4-sad-v1+ #3
> [46799.053166] Call Trace:
> [46799.053180] [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46799.053184] [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46799.053190] [<ffffffff810ab4e9>] ? __trace_bprintk+0x48/0x4a
> [46799.053194] [<ffffffff812097a5>] __list_del_entry+0x8d/0x98
> [46799.053198] [<ffffffff812097be>] list_del+0xe/0x2e
> [46799.053220] [<ffffffffa017b2f5>]
> btrfs_clean_old_snapshots+0xed/0x150 [btrfs]
> [46799.053235] [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46799.053249] [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46799.053254] [<ffffffff8104c750>] kthread+0xba/0xc2
> [46799.053259] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46799.053264] [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46799.053269] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46799.053272] ---[ end trace 7528086f91b151b6 ]---
> [46811.162649] ------------[ cut here ]------------
> [46811.162665] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
> [46811.162669] Hardware name: OptiPlex 745
> [46811.162674] list_del corruption, ffff880050c27c38->next is
> LIST_POISON1 (dead000000100100)
> [46811.162678] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46811.162750] Pid: 18210, comm: btrfs-cleaner Tainted: G W
> 3.7.4-sad-v1+ #3
> [46811.162754] Call Trace:
> [46811.162764] [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46811.162771] [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46811.162779] [<ffffffff810ab4e9>] ? __trace_bprintk+0x48/0x4a
> [46811.162785] [<ffffffff812097a5>] __list_del_entry+0x8d/0x98
> [46811.162791] [<ffffffff812097be>] list_del+0xe/0x2e
> [46811.162820] [<ffffffffa017b2f5>]
> btrfs_clean_old_snapshots+0xed/0x150 [btrfs]
> [46811.162841] [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46811.162862] [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46811.162869] [<ffffffff8104c750>] kthread+0xba/0xc2
> [46811.162875] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.162882] [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46811.162888] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.162892] ---[ end trace 7528086f91b151b7 ]---
> [46811.162904] BUG: unable to handle kernel paging request at 0000000047c5a000
> [46811.163003] IP: [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003] PGD 0
> [46811.163003] Oops: 0000 [#1] SMP
> [46811.163003] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46811.163003] CPU 0
> [46811.163003] Pid: 18210, comm: btrfs-cleaner Tainted: G W
> 3.7.4-sad-v1+ #3 Dell Inc. OptiPlex 745
> /0WF810
> [46811.163003] RIP: 0010:[<ffffffffa017b30b>] [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003] RSP: 0018:ffff8800057fde38 EFLAGS: 00010296
> [46811.163003] RAX: 0000000047c5a000 RBX: ffff880050c27800 RCX: 0000000000000008
> [46811.163003] RDX: 0000000000000000 RSI: ffff8800057fdd70 RDI: ffff880050c27c70
> [46811.163003] RBP: ffff8800057fde78 R08: 0000000000000000 R09: 0000000000000283
> [46811.163003] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880000000000
> [46811.163003] R13: 0000160000000000 R14: ffff8800057fde38 R15: ffff880050c27c38
> [46811.163003] FS: 0000000000000000(0000) GS:ffff88007f200000(0000)
> knlGS:0000000000000000
> [46811.163003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [46811.163003] CR2: 0000000047c5a000 CR3: 000000003f270000 CR4: 00000000000007f0
> [46811.163003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [46811.163003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [46811.163003] Process btrfs-cleaner (pid: 18210, threadinfo
> ffff8800057fc000, task ffff88007c030d40)
> [46811.163003] Stack:
> [46811.163003] ffff880050c27c38 ffff88001f488438 ffff8800057fde48
> ffff88002d15b800
> [46811.163003] ffff88007c030d40 ffff88007c030d40 ffff88007c030d40
> 0000000000000000
> [46811.163003] ffff8800057fdeb8 ffffffffa0173d7d ffff88002d15b800
> 0000000000000000
> [46811.163003] Call Trace:
> [46811.163003] [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46811.163003] [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46811.163003] [<ffffffff8104c750>] kthread+0xba/0xc2
> [46811.163003] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.163003] [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46811.163003] [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.163003] Code: c7 c7 d5 b2 17 a0 31 c0 e8 b4 01 f3 e0 4c 89 ff
> e8 bb e4 08 e1 48 89 df e8 f2 f5 03 00 49 8b 87 c8 fb ff ff 48 8b 80
> 50 01 00 00 <48> 8b 00 4c 01 e8 48 c1 f8 06 48 c1 e0 0c 42 0f b6 44 20
> 3f 31
> [46811.163003] RIP [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003] RSP <ffff8800057fde38>
> [46811.163003] CR2: 0000000047c5a000
> [46811.238512] ---[ end trace 7528086f91b151b8 ]---
next prev parent reply other threads:[~2013-01-25 15:45 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-01-16 12:36 [PATCH V5] Btrfs: snapshot-aware defrag Liu Bo
2013-01-17 14:42 ` Mitch Harder
2013-01-18 0:53 ` Liu Bo
2013-01-18 5:23 ` Mitch Harder
2013-01-18 12:19 ` David Sterba
2013-01-18 22:01 ` Mitch Harder
2013-01-22 17:41 ` Mitch Harder
2013-01-23 7:51 ` Liu Bo
2013-01-23 16:05 ` Mitch Harder
2013-01-24 0:52 ` Liu Bo
2013-01-25 14:55 ` Mitch Harder
2013-01-25 15:40 ` Stefan Behrens
2013-01-27 13:19 ` Liu Bo
2013-01-28 16:55 ` Stefan Behrens
2013-02-16 6:47 ` Liu Bo
2013-02-18 16:53 ` Stefan Behrens
2013-02-19 4:29 ` Liu Bo
2013-02-19 17:53 ` Stefan Behrens
2013-01-25 15:42 ` Liu Bo [this message]
2013-01-25 18:16 ` Mitch Harder
2013-01-27 12:41 ` Liu Bo
2013-01-28 5:20 ` Mitch Harder
2013-01-28 6:54 ` Liu Bo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130125154236.GA3997@liubo \
--to=bo.li.liu@oracle.com \
--cc=JBacik@fusionio.com \
--cc=chris.mason@fusionio.com \
--cc=dave@jikos.cz \
--cc=kitayama@cl.bb4u.ne.jp \
--cc=linux-btrfs@vger.kernel.org \
--cc=miaox@cn.fujitsu.com \
--cc=mitch.harder@sabayonlinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).