WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Martin Mailand]

Hi
I am hitting this Warning reproducible, the workload is a ceph osd, 
kernel ist 3.1.0-rc5.

Best Regards,
  martin

[ 5472.099766] ------------[ cut here ]------------
[ 5472.099833] WARNING: at fs/btrfs/inode.c:2193 
btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
[ 5472.099838] Hardware name: MS-96B3
[ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm 
i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp 
i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs 
zlib_deflate libcrc32c
[ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
[ 5472.099882] Call Trace:
[ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
[ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
[ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0 
[btrfs]
[ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0 
[btrfs]
[ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
[ 5472.099997]  [<ffffffffa003966f>] 
btrfs_commit_transaction+0x3ef/0x870 [btrfs]
[ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
[ 5472.100120]  [<ffffffffa0039af0>] ? 
btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
[ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
[ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
[ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
[ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
[ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
[ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
[ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Liu Bo]

On 09/11/2011 05:47 AM, Martin Mailand wrote:
> Hi
> I am hitting this Warning reproducible, the workload is a ceph osd,
> kernel ist 3.1.0-rc5.
> 

Have posted a patch for this:

http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

thanks,
liubo

> Best Regards,
>  martin
> 
> [ 5472.099766] ------------[ cut here ]------------
> [ 5472.099833] WARNING: at fs/btrfs/inode.c:2193
> btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
> [ 5472.099838] Hardware name: MS-96B3
> [ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm
> i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp
> i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs
> zlib_deflate libcrc32c
> [ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
> [ 5472.099882] Call Trace:
> [ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
> [ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
> [ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0
> [btrfs]
> [ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0
> [btrfs]
> [ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
> [ 5472.099997]  [<ffffffffa003966f>]
> btrfs_commit_transaction+0x3ef/0x870 [btrfs]
> [ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
> [ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
> [ 5472.100120]  [<ffffffffa0039af0>] ?
> btrfs_commit_transaction+0x870/0x870 [btrfs]
> [ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
> [ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
> [ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
> [ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
> [ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
> [ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
> [ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
> [ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
> [ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Sage Weil]

On Tue, 13 Sep 2011, Liu Bo wrote:
> On 09/11/2011 05:47 AM, Martin Mailand wrote:
> > Hi
> > I am hitting this Warning reproducible, the workload is a ceph osd,
> > kernel ist 3.1.0-rc5.
> > 
> 
> Have posted a patch for this:
> 
> http://marc.info/?l=linux-btrfs&m=131547325515336&w=2

We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

I haven't looked at the reservation code in much detail.  Is there 
anything I can do to help track this down?

Thanks-
sage


> 
> thanks,
> liubo
> 
> > Best Regards,
> >  martin
> > 
> > [ 5472.099766] ------------[ cut here ]------------
> > [ 5472.099833] WARNING: at fs/btrfs/inode.c:2193
> > btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()
> > [ 5472.099838] Hardware name: MS-96B3
> > [ 5472.099842] Modules linked in: radeon ttm drm_kms_helper drm
> > i2c_algo_bit psmouse sp5100_tco edac_core lp shpchp serio_raw k8temp
> > i2c_piix4 edac_mce_amd parport ahci pata_atiixp e1000e libahci btrfs
> > zlib_deflate libcrc32c
> > [ 5472.099878] Pid: 2066, comm: kworker/1:1 Not tainted 3.1.0-rc5-custom #1
> > [ 5472.099882] Call Trace:
> > [ 5472.099898]  [<ffffffff81063c1f>] warn_slowpath_common+0x7f/0xc0
> > [ 5472.099907]  [<ffffffff81063c7a>] warn_slowpath_null+0x1a/0x20
> > [ 5472.099935]  [<ffffffffa003f420>] btrfs_orphan_commit_root+0xb0/0xc0
> > [btrfs]
> > [ 5472.099961]  [<ffffffffa00380aa>] commit_fs_roots.clone.21+0xba/0x1a0
> > [btrfs]
> > [ 5472.099971]  [<ffffffff815db96e>] ? _raw_spin_lock+0xe/0x20
> > [ 5472.099997]  [<ffffffffa003966f>]
> > btrfs_commit_transaction+0x3ef/0x870 [btrfs]
> > [ 5472.100065]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
> > [ 5472.100084]  [<ffffffff81086bf0>] ? wake_up_bit+0x40/0x40
> > [ 5472.100120]  [<ffffffffa0039af0>] ?
> > btrfs_commit_transaction+0x870/0x870 [btrfs]
> > [ 5472.100155]  [<ffffffffa0039b0f>] do_async_commit+0x1f/0x30 [btrfs]
> > [ 5472.100171]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
> > [ 5472.100187]  [<ffffffff81081c69>] worker_thread+0x169/0x360
> > [ 5472.100203]  [<ffffffff81081b00>] ? manage_workers.clone.21+0x240/0x240
> > [ 5472.100220]  [<ffffffff81086496>] kthread+0x96/0xa0
> > [ 5472.100236]  [<ffffffff815e5bb4>] kernel_thread_helper+0x4/0x10
> > [ 5472.100253]  [<ffffffff81086400>] ? flush_kthread_worker+0xb0/0xb0
> > [ 5472.100269]  [<ffffffff815e5bb0>] ? gs_change+0x13/0x13
> > [ 5472.100279] ---[ end trace a8bae5767c2c3e55 ]---
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Josef Bacik]

On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
> On Tue, 13 Sep 2011, Liu Bo wrote:
> > On 09/11/2011 05:47 AM, Martin Mailand wrote:
> > > Hi
> > > I am hitting this Warning reproducible, the workload is a ceph osd,
> > > kernel ist 3.1.0-rc5.
> > > 
> > 
> > Have posted a patch for this:
> > 
> > http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
> 
> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
> 
> I haven't looked at the reservation code in much detail.  Is there 
> anything I can do to help track this down?
> 

This should be taken care of with all my enospc changes.  You can pull them down
from my btrfs-work tree as soon as kernel.org comes back from the dead :).
Thanks,

Josef

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[David Sterba]

On Thu, Sep 15, 2011 at 03:50:29PM -0400, Josef Bacik wrote:
> > We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
> > 
> > I haven't looked at the reservation code in much detail.  Is there 
> > anything I can do to help track this down?
> > 
> 
> This should be taken care of with all my enospc changes.  You can pull them down
> from my btrfs-work tree as soon as kernel.org comes back from the dead :).

should you need it earlier, here's a copy:

git://repo.or.cz/linux-2.6/btrfs-unstable.git #git.kernel.org/josef/master


david

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Sage Weil]

On Thu, 15 Sep 2011, David Sterba wrote:
> On Thu, Sep 15, 2011 at 03:50:29PM -0400, Josef Bacik wrote:
> > > We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
> > > 
> > > I haven't looked at the reservation code in much detail.  Is there 
> > > anything I can do to help track this down?
> > > 
> > 
> > This should be taken care of with all my enospc changes.  You can pull them down
> > from my btrfs-work tree as soon as kernel.org comes back from the dead :).
> 
> should you need it earlier, here's a copy:
> 
> git://repo.or.cz/linux-2.6/btrfs-unstable.git #git.kernel.org/josef/master

Thanks!  We'll do some testing today and see if it behaves better.  :)

sage

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Martin Mailand]

Hi Josef,
after a quick test it seems that I do not hit this Warning any longer.
But I got a new one.

[ 5241.839951] ------------[ cut here ]------------
[ 5241.839974] WARNING: at fs/btrfs/extent-tree.c:5715 
btrfs_alloc_free_block+0xac/0x330 [btrfs]()
[ 5241.839979] Hardware name: MS-96B3
[ 5241.839982] Modules linked in: radeon ttm drm_kms_helper drm 
i2c_algo_bit psmouse k8temp sp5100_tco edac_core edac_mce_amd serio_raw 
shpchp i2c_piix4 lp parport ahci pata_atiixp libahci btrfs e1000e 
zlib_deflate libcrc32c
[ 5241.840068] Pid: 1568, comm: kworker/0:1 Tainted: G        W 
3.1.0-rc6 #1
[ 5241.840072] Call Trace:
[ 5241.840084]  [<ffffffff81063d0f>] warn_slowpath_common+0x7f/0xc0
[ 5241.840101]  [<ffffffff81063d6a>] warn_slowpath_null+0x1a/0x20
[ 5241.840133]  [<ffffffffa002a9cc>] btrfs_alloc_free_block+0xac/0x330 
[btrfs]
[ 5241.840152]  [<ffffffff8110d35a>] ? unlock_page+0x2a/0x40
[ 5241.840188]  [<ffffffffa0059268>] ? read_extent_buffer+0xa8/0x180 [btrfs]
[ 5241.840222]  [<ffffffffa0031c00>] ? verify_parent_transid+0x160/0x160 
[btrfs]
[ 5241.840252]  [<ffffffffa001a0d2>] __btrfs_cow_block+0x122/0x4b0 [btrfs]
[ 5241.840283]  [<ffffffffa001a552>] btrfs_cow_block+0xf2/0x1f0 [btrfs]
[ 5241.840314]  [<ffffffffa001cb88>] push_leaf_left+0x108/0x180 [btrfs]
[ 5241.840344]  [<ffffffffa001fb78>] btrfs_del_items+0x2b8/0x440 [btrfs]
[ 5241.840379]  [<ffffffffa00300c2>] btrfs_del_csums+0x2d2/0x310 [btrfs]
[ 5241.840415]  [<ffffffffa00677a8>] ? btrfs_tree_unlock+0x28/0xb0 [btrfs]
[ 5241.840447]  [<ffffffffa002597a>] __btrfs_free_extent+0x48a/0x6f0 [btrfs]
[ 5241.840480]  [<ffffffffa0028c8d>] run_clustered_refs+0x21d/0x840 [btrfs]
[ 5241.840514]  [<ffffffffa002937a>] btrfs_run_delayed_refs+0xca/0x220 
[btrfs]
[ 5241.840551]  [<ffffffffa0053576>] ? 
btrfs_run_ordered_operations+0x1d6/0x200 [btrfs]
[ 5241.840587]  [<ffffffffa0038fa3>] btrfs_commit_transaction+0x83/0x870 
[btrfs]
[ 5241.840605]  [<ffffffff81012871>] ? __switch_to+0x261/0x2f0
[ 5241.840622]  [<ffffffff81086d70>] ? wake_up_bit+0x40/0x40
[ 5241.840656]  [<ffffffffa0039790>] ? 
btrfs_commit_transaction+0x870/0x870 [btrfs]
[ 5241.840691]  [<ffffffffa00397af>] do_async_commit+0x1f/0x30 [btrfs]
[ 5241.840708]  [<ffffffff8108110d>] process_one_work+0x11d/0x430
[ 5241.840724]  [<ffffffff81081dd9>] worker_thread+0x169/0x360
[ 5241.840741]  [<ffffffff81081c70>] ? manage_workers.clone.21+0x240/0x240
[ 5241.840758]  [<ffffffff81086616>] kthread+0x96/0xa0
[ 5241.840775]  [<ffffffff815f2434>] kernel_thread_helper+0x4/0x10
[ 5241.840792]  [<ffffffff81086580>] ? flush_kthread_worker+0xb0/0xb0
[ 5241.840808]  [<ffffffff815f2430>] ? gs_change+0x13/0x13
[ 5241.840819] ---[ end trace c8a580615cad6cb5 ]---


Best Regards,
  Martin

Am 15.09.2011 21:50, schrieb Josef Bacik:
> On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
>> On Tue, 13 Sep 2011, Liu Bo wrote:
>>> On 09/11/2011 05:47 AM, Martin Mailand wrote:
>>>> Hi
>>>> I am hitting this Warning reproducible, the workload is a ceph osd,
>>>> kernel ist 3.1.0-rc5.
>>>>
>>>
>>> Have posted a patch for this:
>>>
>>> http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
>>
>> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
>>
>> I haven't looked at the reservation code in much detail.  Is there
>> anything I can do to help track this down?
>>
>
> This should be taken care of with all my enospc changes.  You can pull them down
> from my btrfs-work tree as soon as kernel.org comes back from the dead :).
> Thanks,
>
> Josef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Josef Bacik]

On 09/16/2011 10:09 AM, Martin Mailand wrote:
> Hi Josef,
> after a quick test it seems that I do not hit this Warning any longer.
> But I got a new one.
> 

Hmm looks like that may not be my newest stuff, is commit

57f499e1bb76ba3ebeb09cd12e9dac84baa5812b

in there?  Specifically look at __btrfs_end_transaction in transaction.c
and see if the line

trans->block_rsv = NULL;

is before the first while() loop.  Thanks,

Josef

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Martin Mailand]

Hi Josef,
the commit is not in there, but the code looks like your post.

if (--trans->use_count) {
                 trans->block_rsv = trans->orig_rsv;
                 return 0;
         }

         trans->block_rsv = NULL;
         while (count < 4) {
                 unsigned long cur = trans->delayed_ref_updates;
                 trans->delayed_ref_updates = 0;

But on the other hand I am quite new to git, how could I get your latest 
commit?

Best Regards,
  Martin

Am 16.09.2011 16:37, schrieb Josef Bacik:
> On 09/16/2011 10:09 AM, Martin Mailand wrote:
>> Hi Josef,
>> after a quick test it seems that I do not hit this Warning any longer.
>> But I got a new one.
>>
>
> Hmm looks like that may not be my newest stuff, is commit
>
> 57f499e1bb76ba3ebeb09cd12e9dac84baa5812b
>
> in there?  Specifically look at __btrfs_end_transaction in transaction.c
> and see if the line
>
> trans->block_rsv = NULL;
>
> is before the first while() loop.  Thanks,
>
> Josef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[David Sterba]

On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
> On Tue, 13 Sep 2011, Liu Bo wrote:
> > On 09/11/2011 05:47 AM, Martin Mailand wrote:
> > > Hi
> > > I am hitting this Warning reproducible, the workload is a ceph osd,
> > > kernel ist 3.1.0-rc5.
> > > 
> > 
> > Have posted a patch for this:
> > 
> > http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
> 
> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.

Me too, for the 

WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 [btrfs]()

case mentioned in the changelog. I optimistically dropped the ratelimit
patch for that WARN_ON, but had to add it quickly back. Unfortunatelly I
do not have a reliable reproducer. It justs starts sometime during
xfstests, maybe after a few rounds.


david

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Jim Schutt]

David Sterba wrote:
> On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
>> On Tue, 13 Sep 2011, Liu Bo wrote:
>>> On 09/11/2011 05:47 AM, Martin Mailand wrote:
>>>> Hi
>>>> I am hitting this Warning reproducible, the workload is a ceph osd,
>>>> kernel ist 3.1.0-rc5.
>>>>
>>> Have posted a patch for this:
>>>
>>> http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
>> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
> 
> Me too, for the 
> 
> WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 [btrfs]()
> 

FWIW, I'm seeing a slightly different case, while testing branch
integration/btrfs-next (commit 2828cbd9620e03) from
git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch
master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git

Under a heavy ceph write load, I see lots of these:

[ 2369.797044] ------------[ cut here ]------------
[ 2369.801759] WARNING: at fs/btrfs/extent-tree.c:5751 use_block_rsv+0x177/0x180 [btrfs]()
[ 2369.809864] Hardware name: X8DTH-i/6/iF/6F
[ 2369.814062] Modules linked in: loop btrfs zlib_deflate lzo_compress ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
[ 2369.828671] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2369.829040]  ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp scsi_transport_iscsi rds ib_ipoib rdma_ucm rdma_cm ib_ucm ib_uverbs ib_umad ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod video sbs sbshc pci_slot battery acpi_pad ac kvm sg sd_mod mlx4_ib ib_mad ib_core mlx4_en joydev mpt2sas tpm_tis tpm scsi_transport_sas mlx4_core cxgb4 button serio_raw raid_class tpm_bios ata_piix libata scsi_mod i2c_i801 ioatdma ehci_hcd iTCO_wdt uhci_hcd i2c_core i7core_edac iTCO_vendor_support edac_core pcspkr rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 igb dca e1000
[ 2369.889908] Pid: 23744, comm: kworker/19:3 Tainted: G        W   3.1.0-rc6-00265-gf883c8c #33
[ 2369.898510] Call Trace:
[ 2369.901026]  [<ffffffff8104e02f>] warn_slowpath_common+0x7f/0xc0
[ 2369.907046]  [<ffffffff8104e08a>] warn_slowpath_null+0x1a/0x20
[ 2369.912935]  [<ffffffffa05c7fb7>] use_block_rsv+0x177/0x180 [btrfs]
[ 2369.919422]  [<ffffffffa05d122d>] btrfs_alloc_free_block+0x3d/0x220 [btrfs]
[ 2369.926431]  [<ffffffff810f1a01>] ? __set_page_dirty_nobuffers+0xe1/0x150
[ 2369.933265]  [<ffffffffa05ff3c1>] ? read_extent_buffer+0xc1/0x1a0 [btrfs]
[ 2369.940085]  [<ffffffffa05bf58e>] __btrfs_cow_block+0x11e/0x4d0 [btrfs]
[ 2369.946737]  [<ffffffffa0610091>] ? btrfs_tree_lock+0x161/0x2c0 [btrfs]
[ 2369.953391]  [<ffffffffa05bfefa>] btrfs_cow_block+0xea/0x200 [btrfs]
[ 2369.959756]  [<ffffffffa05c2b2f>] btrfs_search_slot+0x31f/0x720 [btrfs]
[ 2369.966362]  [<ffffffffa05d3a9b>] btrfs_del_csums+0x23b/0x320 [btrfs]
[ 2369.972833]  [<ffffffffa05cf41b>] __btrfs_free_extent+0x49b/0x720 [btrfs]
[ 2369.979660]  [<ffffffffa05cf969>] run_delayed_data_ref+0x159/0x160 [btrfs]
[ 2369.986591]  [<ffffffffa05cfa20>] run_one_delayed_ref+0xb0/0xd0 [btrfs]
[ 2369.993257]  [<ffffffffa05cfb0f>] run_clustered_refs+0xcf/0x240 [btrfs]
[ 2369.999928]  [<ffffffffa05cfd58>] btrfs_run_delayed_refs+0xd8/0x260 [btrfs]
[ 2370.006937]  [<ffffffffa05dde77>] btrfs_commit_transaction+0x87/0x8b0 [btrfs]
[ 2370.014238]  [<ffffffff81072930>] ? wake_up_bit+0x40/0x40
[ 2370.019710]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.027180]  [<ffffffffa05de6bf>] do_async_commit+0x1f/0x30 [btrfs]
[ 2370.033578]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.041039]  [<ffffffff8106c03f>] process_one_work+0x13f/0x490
[ 2370.046950]  [<ffffffff8106ddb7>] worker_thread+0x187/0x3e0
[ 2370.052679]  [<ffffffff8106dc30>] ? manage_workers+0x120/0x120
[ 2370.058682]  [<ffffffff810723a6>] kthread+0x96/0xa0
[ 2370.063690]  [<ffffffff8145c7f4>] kernel_thread_helper+0x4/0x10
[ 2370.069695]  [<ffffffff8145284a>] ? retint_restore_args+0xe/0xe
[ 2370.075720]  [<ffffffff81072310>] ? kthread_worker_fn+0x1d0/0x1d0
[ 2370.081919]  [<ffffffff8145c7f0>] ? gs_change+0xb/0xb
[ 2370.087103] ---[ end trace a6d5cd679d4e46b9 ]---


I don't know if it matters, but I also see lots of this sort of thing:

[ 2370.131721] btrfs: all snaps cleaned
[ 2370.241382] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.275303] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.275306] btrfs: all snaps cleaned
[ 2370.296461] btrfs: all snaps cleaned
[ 2370.639211] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.688785] btrfs: all snaps cleaned
[ 2370.811568] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2370.830958] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.830962] btrfs: all snaps cleaned
[ 2371.040634] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.057286] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.267641] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.285724] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.301746] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.328422] btrfs: all snaps cleaned
[ 2371.371326] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.417955] btrfs: all snaps cleaned
[ 2371.498012] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.530904] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2371.530906] btrfs: all snaps cleaned
[ 2371.553309] btrfs: all snaps cleaned
[ 2371.574753] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.625655] btrfs: all snaps cleaned

-- Jim

> 
> 
> david
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Jim Schutt]

David Sterba wrote:
> On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
>> On Tue, 13 Sep 2011, Liu Bo wrote:
>>> On 09/11/2011 05:47 AM, Martin Mailand wrote:
>>>> Hi
>>>> I am hitting this Warning reproducible, the workload is a ceph osd,
>>>> kernel ist 3.1.0-rc5.
>>>>
>>> Have posted a patch for this:
>>>
>>> http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
>> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
> 
> Me too, for the 
> 
> WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 [btrfs]()
> 

FWIW, I'm seeing a slightly different case, while testing branch
integration/btrfs-next (commit 2828cbd9620e03) from
git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch
master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git

Under a heavy ceph write load, I see lots of these:

[ 2369.797044] ------------[ cut here ]------------
[ 2369.801759] WARNING: at fs/btrfs/extent-tree.c:5751 use_block_rsv+0x177/0x180 [btrfs]()
[ 2369.809864] Hardware name: X8DTH-i/6/iF/6F
[ 2369.814062] Modules linked in: loop btrfs zlib_deflate lzo_compress ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack
[ 2369.828671] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2369.829040]  ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp scsi_transport_iscsi rds ib_ipoib rdma_ucm rdma_cm ib_ucm ib_uverbs ib_umad ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash dm_log dm_multipath scsi_dh dm_mod video sbs sbshc pci_slot battery acpi_pad ac kvm sg sd_mod mlx4_ib ib_mad ib_core mlx4_en joydev mpt2sas tpm_tis tpm scsi_transport_sas mlx4_core cxgb4 button serio_raw raid_class tpm_bios ata_piix libata scsi_mod i2c_i801 ioatdma ehci_hcd iTCO_wdt uhci_hcd i2c_core i7core_edac iTCO_vendor_support edac_core pcspkr rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 igb dca e1000
[ 2369.889908] Pid: 23744, comm: kworker/19:3 Tainted: G        W   3.1.0-rc6-00265-gf883c8c #33
[ 2369.898510] Call Trace:
[ 2369.901026]  [<ffffffff8104e02f>] warn_slowpath_common+0x7f/0xc0
[ 2369.907046]  [<ffffffff8104e08a>] warn_slowpath_null+0x1a/0x20
[ 2369.912935]  [<ffffffffa05c7fb7>] use_block_rsv+0x177/0x180 [btrfs]
[ 2369.919422]  [<ffffffffa05d122d>] btrfs_alloc_free_block+0x3d/0x220 [btrfs]
[ 2369.926431]  [<ffffffff810f1a01>] ? __set_page_dirty_nobuffers+0xe1/0x150
[ 2369.933265]  [<ffffffffa05ff3c1>] ? read_extent_buffer+0xc1/0x1a0 [btrfs]
[ 2369.940085]  [<ffffffffa05bf58e>] __btrfs_cow_block+0x11e/0x4d0 [btrfs]
[ 2369.946737]  [<ffffffffa0610091>] ? btrfs_tree_lock+0x161/0x2c0 [btrfs]
[ 2369.953391]  [<ffffffffa05bfefa>] btrfs_cow_block+0xea/0x200 [btrfs]
[ 2369.959756]  [<ffffffffa05c2b2f>] btrfs_search_slot+0x31f/0x720 [btrfs]
[ 2369.966362]  [<ffffffffa05d3a9b>] btrfs_del_csums+0x23b/0x320 [btrfs]
[ 2369.972833]  [<ffffffffa05cf41b>] __btrfs_free_extent+0x49b/0x720 [btrfs]
[ 2369.979660]  [<ffffffffa05cf969>] run_delayed_data_ref+0x159/0x160 [btrfs]
[ 2369.986591]  [<ffffffffa05cfa20>] run_one_delayed_ref+0xb0/0xd0 [btrfs]
[ 2369.993257]  [<ffffffffa05cfb0f>] run_clustered_refs+0xcf/0x240 [btrfs]
[ 2369.999928]  [<ffffffffa05cfd58>] btrfs_run_delayed_refs+0xd8/0x260 [btrfs]
[ 2370.006937]  [<ffffffffa05dde77>] btrfs_commit_transaction+0x87/0x8b0 [btrfs]
[ 2370.014238]  [<ffffffff81072930>] ? wake_up_bit+0x40/0x40
[ 2370.019710]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.027180]  [<ffffffffa05de6bf>] do_async_commit+0x1f/0x30 [btrfs]
[ 2370.033578]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 [btrfs]
[ 2370.041039]  [<ffffffff8106c03f>] process_one_work+0x13f/0x490
[ 2370.046950]  [<ffffffff8106ddb7>] worker_thread+0x187/0x3e0
[ 2370.052679]  [<ffffffff8106dc30>] ? manage_workers+0x120/0x120
[ 2370.058682]  [<ffffffff810723a6>] kthread+0x96/0xa0
[ 2370.063690]  [<ffffffff8145c7f4>] kernel_thread_helper+0x4/0x10
[ 2370.069695]  [<ffffffff8145284a>] ? retint_restore_args+0xe/0xe
[ 2370.075720]  [<ffffffff81072310>] ? kthread_worker_fn+0x1d0/0x1d0
[ 2370.081919]  [<ffffffff8145c7f0>] ? gs_change+0xb/0xb
[ 2370.087103] ---[ end trace a6d5cd679d4e46b9 ]---


I don't know if it matters, but I also see lots of this sort of thing:

[ 2370.131721] btrfs: all snaps cleaned
[ 2370.241382] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.275303] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.275306] btrfs: all snaps cleaned
[ 2370.296461] btrfs: all snaps cleaned
[ 2370.639211] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2370.688785] btrfs: all snaps cleaned
[ 2370.811568] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2370.830958] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2370.830962] btrfs: all snaps cleaned
[ 2371.040634] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.057286] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.267641] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.285724] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.301746] btrfs: journal_info set but trans not join/nolock: 0x1
[ 2371.328422] btrfs: all snaps cleaned
[ 2371.371326] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.417955] btrfs: all snaps cleaned
[ 2371.498012] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.530904] btrfs: btrfs_clean_old_snapshots to process 0 old snaps
[ 2371.530906] btrfs: all snaps cleaned
[ 2371.553309] btrfs: all snaps cleaned
[ 2371.574753] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
[ 2371.625655] btrfs: all snaps cleaned

-- Jim

> 
> 
> david
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[David Sterba]

On Fri, Sep 16, 2011 at 10:25:51AM -0600, Jim Schutt wrote:
> >WARNING: at fs/btrfs/extent-tree.c:5711 btrfs_alloc_free_block+0x180/0x350 
> >[btrfs]()
> >
> 
> FWIW, I'm seeing a slightly different case, while testing branch
> integration/btrfs-next (commit 2828cbd9620e03) from
> git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch
> master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git
> 
> Under a heavy ceph write load, I see lots of these:

same warning from mainline fs/btrfs/extent-tree.c:5711 matching
fs/btrfs/extent-tree.c:5751 in the btrfs-next branch and is ratelimited.
It does not seem to be fixed by the patch, but as the branch contains
other patches, it could be caused ty them, I didn't have a closer look.
You can try with just Josef's branch on top of mainline if you have a
good way to trigger the warnings.

> [ 2369.797044] ------------[ cut here ]------------
> [ 2369.801759] WARNING: at fs/btrfs/extent-tree.c:5751 
> use_block_rsv+0x177/0x180 [btrfs]()
> [ 2369.809864] Hardware name: X8DTH-i/6/iF/6F
> [ 2369.814062] Modules linked in: loop btrfs zlib_deflate lzo_compress 
> ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state 
> nf_conntrack
> [ 2369.828671] btrfs: journal_info set but trans not join/nolock: 0x1
> [ 2369.829040]  ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables 
> bridge stp scsi_transport_iscsi rds ib_ipoib rdma_ucm rdma_cm ib_ucm 
> ib_uverbs ib_umad ib_cm iw_cm ib_addr ipv6 ib_sa dm_mirror dm_region_hash 
> dm_log dm_multipath scsi_dh dm_mod video sbs sbshc pci_slot battery 
> acpi_pad ac kvm sg sd_mod mlx4_ib ib_mad ib_core mlx4_en joydev mpt2sas 
> tpm_tis tpm scsi_transport_sas mlx4_core cxgb4 button serio_raw raid_class 
> tpm_bios ata_piix libata scsi_mod i2c_i801 ioatdma ehci_hcd iTCO_wdt 
> uhci_hcd i2c_core i7core_edac iTCO_vendor_support edac_core pcspkr rtc nfs 
> nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 igb dca e1000
> [ 2369.889908] Pid: 23744, comm: kworker/19:3 Tainted: G        W   
> 3.1.0-rc6-00265-gf883c8c #33
> [ 2369.898510] Call Trace:
> [ 2369.901026]  [<ffffffff8104e02f>] warn_slowpath_common+0x7f/0xc0
> [ 2369.907046]  [<ffffffff8104e08a>] warn_slowpath_null+0x1a/0x20
> [ 2369.912935]  [<ffffffffa05c7fb7>] use_block_rsv+0x177/0x180 [btrfs]
> [ 2369.919422]  [<ffffffffa05d122d>] btrfs_alloc_free_block+0x3d/0x220 
> [btrfs]
> [ 2369.926431]  [<ffffffff810f1a01>] ? __set_page_dirty_nobuffers+0xe1/0x150
> [ 2369.933265]  [<ffffffffa05ff3c1>] ? read_extent_buffer+0xc1/0x1a0 [btrfs]
> [ 2369.940085]  [<ffffffffa05bf58e>] __btrfs_cow_block+0x11e/0x4d0 [btrfs]
> [ 2369.946737]  [<ffffffffa0610091>] ? btrfs_tree_lock+0x161/0x2c0 [btrfs]
> [ 2369.953391]  [<ffffffffa05bfefa>] btrfs_cow_block+0xea/0x200 [btrfs]
> [ 2369.959756]  [<ffffffffa05c2b2f>] btrfs_search_slot+0x31f/0x720 [btrfs]
> [ 2369.966362]  [<ffffffffa05d3a9b>] btrfs_del_csums+0x23b/0x320 [btrfs]
> [ 2369.972833]  [<ffffffffa05cf41b>] __btrfs_free_extent+0x49b/0x720 [btrfs]
> [ 2369.979660]  [<ffffffffa05cf969>] run_delayed_data_ref+0x159/0x160 
> [btrfs]
> [ 2369.986591]  [<ffffffffa05cfa20>] run_one_delayed_ref+0xb0/0xd0 [btrfs]
> [ 2369.993257]  [<ffffffffa05cfb0f>] run_clustered_refs+0xcf/0x240 [btrfs]
> [ 2369.999928]  [<ffffffffa05cfd58>] btrfs_run_delayed_refs+0xd8/0x260 
> [btrfs]
> [ 2370.006937]  [<ffffffffa05dde77>] btrfs_commit_transaction+0x87/0x8b0 
> [btrfs]
> [ 2370.014238]  [<ffffffff81072930>] ? wake_up_bit+0x40/0x40
> [ 2370.019710]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 
> [btrfs]
> [ 2370.027180]  [<ffffffffa05de6bf>] do_async_commit+0x1f/0x30 [btrfs]
> [ 2370.033578]  [<ffffffffa05de6a0>] ? btrfs_commit_transaction+0x8b0/0x8b0 
> [btrfs]
> [ 2370.041039]  [<ffffffff8106c03f>] process_one_work+0x13f/0x490
> [ 2370.046950]  [<ffffffff8106ddb7>] worker_thread+0x187/0x3e0
> [ 2370.052679]  [<ffffffff8106dc30>] ? manage_workers+0x120/0x120
> [ 2370.058682]  [<ffffffff810723a6>] kthread+0x96/0xa0
> [ 2370.063690]  [<ffffffff8145c7f4>] kernel_thread_helper+0x4/0x10
> [ 2370.069695]  [<ffffffff8145284a>] ? retint_restore_args+0xe/0xe
> [ 2370.075720]  [<ffffffff81072310>] ? kthread_worker_fn+0x1d0/0x1d0
> [ 2370.081919]  [<ffffffff8145c7f0>] ? gs_change+0xb/0xb
> [ 2370.087103] ---[ end trace a6d5cd679d4e46b9 ]---
> 
> 
> I don't know if it matters, but I also see lots of this sort of thing:
> 
> [ 2370.131721] btrfs: all snaps cleaned
> [ 2370.241382] btrfs: btrfs_clean_old_snapshots to process 1 old snaps
> [ 2370.275303] btrfs: btrfs_clean_old_snapshots to process 0 old snaps

Harmless debugging of

> [ 2370.275306] btrfs: all snaps cleaned
> [ 2370.296461] btrfs: all snaps cleaned
> [ 2370.639211] btrfs: btrfs_clean_old_snapshots to process 1 old snaps

... slow snapshot deletion reported by BCrook.

> [ 2370.688785] btrfs: all snaps cleaned
> [ 2370.811568] btrfs: journal_info set but trans not join/nolock: 0x1

... btrfs/transaction.c:264 in start_transaction, but the patch was not
correct and reported not only the WARN_ON case.

Thanks for the report, I'll clean the branch a bit and move the
debugging or too experimental stuff into a separate branch.


david

Re: WARNING: at fs/btrfs/inode.c:2193 btrfs_orphan_commit_root+0xb0/0xc0 [btrfs]()

[Josef Bacik]

On 09/16/2011 12:25 PM, Jim Schutt wrote:
> David Sterba wrote:
>> On Thu, Sep 15, 2011 at 11:44:09AM -0700, Sage Weil wrote:
>>> On Tue, 13 Sep 2011, Liu Bo wrote:
>>>> On 09/11/2011 05:47 AM, Martin Mailand wrote:
>>>>> Hi
>>>>> I am hitting this Warning reproducible, the workload is a ceph osd,
>>>>> kernel ist 3.1.0-rc5.
>>>>>
>>>> Have posted a patch for this:
>>>>
>>>> http://marc.info/?l=linux-btrfs&m=131547325515336&w=2
>>> We're still seeing this with -rc6, which includes 98c9942 and 65450aa.
>>
>> Me too, for the
>> WARNING: at fs/btrfs/extent-tree.c:5711
>> btrfs_alloc_free_block+0x180/0x350 [btrfs]()
>>
> 
> FWIW, I'm seeing a slightly different case, while testing branch
> integration/btrfs-next (commit 2828cbd9620e03) from
> git://repo.or.cz/linux-2.6/btrfs-unstable.git merged into branch
> master (commit c455ea4f122d21) from git://github.com/torvalds/linux.git
> 

Ah yeah sorry I see what's going on here, I just missed a few places we
call run_delayed_refs() where we can still have trans->block_rsv set.  I
will fix this and send a patch soon.  Thanks,

Josef