FS / Kernel question choosing the correct kernel version

FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Hi,

i got stuck while selecting the right FS for ceph / RBD.

XFS:
- deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
- under 3.5-rc3 all my machines got loaded doing nothing than waiting 
for XFS / SSDs so ceph is really slow / unuseable

btrfs:
- 3.5-rc3 ceph is pretty fast and works good until i see also a deadlock 
while doing heavy random I/Os in my rbd / kvm.

All processes hang in btrfs_commit_transaction or 
btrfs_commit_transaction_async

Are there tested / recommanded kernel versions for rbd and a specific fs.

Greets
Stefan

Re: FS / Kernel question choosing the correct kernel version

[Sage Weil]

On Sat, 23 Jun 2012, Stefan Priebe wrote:
> Hi,
> 
> i got stuck while selecting the right FS for ceph / RBD.
> 
> XFS:
> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs

There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla 
3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.

> - under 3.5-rc3 all my machines got loaded doing nothing than waiting for XFS
> / SSDs so ceph is really slow / unuseable

> btrfs:
> - 3.5-rc3 ceph is pretty fast and works good until i see also a deadlock while
> doing heavy random I/Os in my rbd / kvm.
> 
> All processes hang in btrfs_commit_transaction or
> btrfs_commit_transaction_async

We haven't seen this yet.  See my other reply; a task dump may offer some 
clues.

> Are there tested / recommanded kernel versions for rbd and a specific fs.

Lockdep noise aside, we've been fine with 3.4 for btrfs and xfs so far.  
Our regression testing hardware is probably not as fast as yours, though, 
which may explain why our qa hasn't hit the same bugs.

Can you be more specific about how you're generating the rbd workload?

sage

Re: FS / Kernel question choosing the correct kernel version

[Christoph Hellwig]

On Mon, Jun 25, 2012 at 03:11:17PM -0700, Sage Weil wrote:
> On Sat, 23 Jun 2012, Stefan Priebe wrote:
> > Hi,
> > 
> > i got stuck while selecting the right FS for ceph / RBD.
> > 
> > XFS:
> > - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
> 
> There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla 
> 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.

Stefan, which deadlock is this, did you report it to the XFS list?

Sage, which lockdep noise?

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Am 26.06.2012 10:14, schrieb Christoph Hellwig:
> On Mon, Jun 25, 2012 at 03:11:17PM -0700, Sage Weil wrote:
>> On Sat, 23 Jun 2012, Stefan Priebe wrote:
>>> Hi,
>>>
>>> i got stuck while selecting the right FS for ceph / RBD.
>>>
>>> XFS:
>>> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
>>
>> There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla
>> 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.
>
> Stefan, which deadlock is this, did you report it to the XFS list?

Yes i did. You are in CC ;-)

http://oss.sgi.com/archives/xfs/2012-05/msg00307.html

But i did not send a sysrq trigger as i then started to work with btrfs. 
As i archieve more than two times better performance with ceph and btrfs.

Stefan

PS: i have this one laying around which is NOT in 3.0.X not sure whether 
this is relevant:

From: Christoph Hellwig <hch@lst.de>
Subject: xfs: don't wait for all pending I/O in ->write_inode

If we wait for all pending I/O in ->write_inode we can starve the caller,
which sine recent changes can also be the flusher thread in kupdate mode.

Fortunately there is no good reason to do the wait, as a blocking caller
already waited for buffered I/O using filemap_write_and_wait_range, and thus
we don't have to rely on this, and kupdated doesn't care for us to finish
the write first, but just wants to snapshot the inode metadata to disk.

Upstream this was fixed in a much more intrusive way by

         xfs: remove i_iocount

and the various patches leading towards it, including changes to the core
AIO code.  I think this simpler patch is the better version for 3.0-stable.

Signed-off-by: Christoph Hellwig <hch@lst.de>

Index: linux-2.6/fs/xfs/linux-2.6/xfs_super.c
===================================================================
--- linux-2.6.orig/fs/xfs/linux-2.6/xfs_super.c 2012-03-18 
09:03:27.583397799 +0100
+++ linux-2.6/fs/xfs/linux-2.6/xfs_super.c      2012-03-18 
09:03:45.083398125 +0100
@@ -892,7 +892,6 @@ xfs_fs_write_inode(
                  * ->sync_fs call do that for thus, which reduces the 
number
                  * of synchronous log foces dramatically.
                  */
-               xfs_ioend_wait(ip);
                 error = xfs_log_dirty_inode(ip, NULL, 0);
                 if (error)
                         goto out;

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Am 26.06.2012 10:14, schrieb Christoph Hellwig:
> On Mon, Jun 25, 2012 at 03:11:17PM -0700, Sage Weil wrote:
>> On Sat, 23 Jun 2012, Stefan Priebe wrote:
>>> Hi,
>>>
>>> i got stuck while selecting the right FS for ceph / RBD.
>>>
>>> XFS:
>>> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
>>
>> There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla
>> 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.
>
> Stefan, which deadlock is this, did you report it to the XFS list?

Yes i did. You are in CC

http://oss.sgi.com/archives/xfs/2012-05/msg00307.html

Stefan

Re: FS / Kernel question choosing the correct kernel version

[Sage Weil]

On Tue, 26 Jun 2012, Christoph Hellwig wrote:
> On Mon, Jun 25, 2012 at 03:11:17PM -0700, Sage Weil wrote:
> > On Sat, 23 Jun 2012, Stefan Priebe wrote:
> > > Hi,
> > > 
> > > i got stuck while selecting the right FS for ceph / RBD.
> > > 
> > > XFS:
> > > - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
> > 
> > There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla 
> > 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.
> 
> Stefan, which deadlock is this, did you report it to the XFS list?
> 
> Sage, which lockdep noise?

There are two versions of the warning at 

	http://tracker.newdream.net/issues/2523

It hit it several times per night; very reproducible.

Thanks!
sage

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Am 26.06.2012 00:11, schrieb Sage Weil:
> On Sat, 23 Jun 2012, Stefan Priebe wrote:
>> Hi,
>>
>> i got stuck while selecting the right FS for ceph / RBD.
>>
>> XFS:
>> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while syncfs
>
> There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla
> 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.

Will test 3.4.4 soon.

>> - under 3.5-rc3 all my machines got loaded doing nothing than waiting for XFS
>> / SSDs so ceph is really slow / unuseable
>
>> btrfs:
>> - 3.5-rc3 ceph is pretty fast and works good until i see also a deadlock while
>> doing heavy random I/Os in my rbd / kvm.
>>
>> All processes hang in btrfs_commit_transaction or
>> btrfs_commit_transaction_async
>
> We haven't seen this yet.  See my other reply; a task dump may offer some
> clues.
You can find the task dump here:
http://pastebin.com/raw.php?i=R4pztDRt

Stefan

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Am 26.06.2012 11:07, schrieb Stefan Priebe:
> Am 26.06.2012 00:11, schrieb Sage Weil:
>> On Sat, 23 Jun 2012, Stefan Priebe wrote:
>>> Hi,
>>>
>>> i got stuck while selecting the right FS for ceph / RBD.
>>>
>>> XFS:
>>> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while
>>> syncfs
>>
>> There was an ilock fix that went into 3.4, IIRC.  Have you tried vanilla
>> 3.4?  We are seeing some lockdep noise currently, but no deadlocks yet.
>
> Will test 3.4.4 soon.

With btrfs i have 13-14.000 iops while doing random 4k writes from kvm. 
With XFS and 3.4.4 i have 814 ;-( and got a lot of slow request noise in 
ceph.log ;-( so 814 iops are not an option to me. Or is there anything 
special i've to respect while using XFS.

Greets
Stefan

Re: FS / Kernel question choosing the correct kernel version

[Mark Nelson]

On 06/26/2012 11:15 AM, Stefan Priebe wrote:
> Am 26.06.2012 11:07, schrieb Stefan Priebe:
>> Am 26.06.2012 00:11, schrieb Sage Weil:
>>> On Sat, 23 Jun 2012, Stefan Priebe wrote:
>>>> Hi,
>>>>
>>>> i got stuck while selecting the right FS for ceph / RBD.
>>>>
>>>> XFS:
>>>> - deadlock / hung task under 3.0.34 in xfs_ilock / xfs_buf_lock while
>>>> syncfs
>>>
>>> There was an ilock fix that went into 3.4, IIRC. Have you tried vanilla
>>> 3.4? We are seeing some lockdep noise currently, but no deadlocks yet.
>>
>> Will test 3.4.4 soon.
>
> With btrfs i have 13-14.000 iops while doing random 4k writes from kvm.
> With XFS and 3.4.4 i have 814 ;-( and got a lot of slow request noise in
> ceph.log ;-( so 814 iops are not an option to me. Or is there anything
> special i've to respect while using XFS.
>
> Greets
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Stefan,

If you can, it would be really interesting to see the blktrace results 
during these tests for both xfs and btrfs.  blktrace is in the ubuntu 
repositories and can be run quite easily from the command line during 
your test.

Thanks,
Mark

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Am 26.06.2012 18:29, schrieb Mark Nelson:
> On 06/26/2012 11:15 AM, Stefan Priebe wrote:
> Hi Stefan,
>
> If you can, it would be really interesting to see the blktrace results
> during these tests for both xfs and btrfs.  blktrace is in the ubuntu
> repositories and can be run quite easily from the command line during
> your test.

Sure any special parameters? I have 4 SSDs per OSD Server.

Or just blktrace -o file?

Stefan

Re: FS / Kernel question choosing the correct kernel version

[Mark Nelson]

On 06/26/2012 11:43 AM, Stefan Priebe wrote:
> Am 26.06.2012 18:29, schrieb Mark Nelson:
>> On 06/26/2012 11:15 AM, Stefan Priebe wrote:
>> Hi Stefan,
>>
>> If you can, it would be really interesting to see the blktrace results
>> during these tests for both xfs and btrfs. blktrace is in the ubuntu
>> repositories and can be run quite easily from the command line during
>> your test.
>
> Sure any special parameters? I have 4 SSDs per OSD Server.
>
> Or just blktrace -o file?
>
> Stefan

For each device you run it on you'll get one file per core.  There may 
be some performance impact if you run blktrace on every device per node. 
  If your data is well distributed, even a trace for one OSD (per test) 
would be interesting.

so blktrace -o <outfile prefix> -d <device> where device is your first 
OSD or something.  If you can do it for both btrfs and xfs and maybe run 
each test for a couple of minutes that might be enough.

Mark

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

I would but both XFS and btrfs are crashing after a short period.

XFS crashes with this one:
[  479.732636] INFO: task ceph-osd:3217 blocked for more than 120 seconds.
[  479.747724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  479.763534] ceph-osd        D ffffffff8180e9c0     0  3217      1 
0x00000000
[  479.779837]  ffff880bc4321bd8 0000000000000082 ffff880bc5694830 
0000000000012200
[  479.779840]  ffff880bc4321fd8 ffff880bc4320010 0000000000012200 
0000000000012200
[  479.779841]  ffff880bc4321fd8 0000000000012200 ffff880e40ea9810 
ffff880bc5694830
[  479.779843] Call Trace:
[  479.779850]  [<ffffffff816296e4>] schedule+0x24/0x70
[  479.779853]  [<ffffffff812c2049>] xlog_wait+0x69/0x90
[  479.779856]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[  479.779858]  [<ffffffff812c23b3>] xlog_cil_push+0x343/0x3c0
[  479.779861]  [<ffffffff8126ce09>] ? xfs_buf_unlock+0x19/0x70
[  479.779862]  [<ffffffff812c2ab1>] xlog_cil_force_lsn+0x101/0x110
[  479.779864]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[  479.779865]  [<ffffffff812bcd77>] ? xfs_trans_free_items+0x87/0xb0
[  479.779867]  [<ffffffff812c07c8>] _xfs_log_force_lsn+0x48/0x290
[  479.779871]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[  479.779872]  [<ffffffff812bdfdb>] xfs_trans_commit+0x24b/0x260
[  479.779875]  [<ffffffff81271e9d>] xfs_fs_log_dummy+0x5d/0x90
[  479.779877]  [<ffffffff812bed9c>] ? xfs_log_need_covered+0x7c/0xc0
[  479.779879]  [<ffffffff8127d378>] xfs_quiesce_data+0x88/0x90
[  479.779881]  [<ffffffff8127b428>] xfs_fs_sync_fs+0x28/0x60
[  479.779884]  [<ffffffff811363ae>] __sync_filesystem+0x5e/0x90
[  479.779885]  [<ffffffff811364b3>] sync_filesystem+0x43/0x60
[  479.779887]  [<ffffffff81136518>] sys_syncfs+0x48/0x80
[  479.779890]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b


Am 26.06.2012 18:59, schrieb Mark Nelson:
> On 06/26/2012 11:43 AM, Stefan Priebe wrote:
>> Am 26.06.2012 18:29, schrieb Mark Nelson:
>>> On 06/26/2012 11:15 AM, Stefan Priebe wrote:
>>> Hi Stefan,
>>>
>>> If you can, it would be really interesting to see the blktrace results
>>> during these tests for both xfs and btrfs. blktrace is in the ubuntu
>>> repositories and can be run quite easily from the command line during
>>> your test.
>>
>> Sure any special parameters? I have 4 SSDs per OSD Server.
>>
>> Or just blktrace -o file?
>>
>> Stefan
>
> For each device you run it on you'll get one file per core.  There may
> be some performance impact if you run blktrace on every device per node.
>   If your data is well distributed, even a trace for one OSD (per test)
> would be interesting.
>
> so blktrace -o <outfile prefix> -d <device> where device is your first
> OSD or something.  If you can do it for both btrfs and xfs and maybe run
> each test for a couple of minutes that might be enough.
>
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

and these:
   359.919732] INFO: task xfsaild/sdd1:2247 blocked for more than 120 
seconds.
[  359.934620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  359.950059] xfsaild/sdd1    D 0000000000000007     0  2247      2 
0x00000000
[  359.966026]  ffff8810349d9cf0 0000000000000046 ffff881034809810 
0000000000012200
[  359.982249]  ffff8810349d9fd8 ffff8810349d8010 0000000000012200 
0000000000012200
[  359.998634]  ffff8810349d9fd8 0000000000012200 ffff881038ef8000 
ffff881034809810
[  360.015139] Call Trace:
[  360.031552]  [<ffffffff816296e4>] schedule+0x24/0x70
[  360.048356]  [<ffffffff812c2a9a>] xlog_cil_force_lsn+0xea/0x110
[  360.065363]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[  360.082348]  [<ffffffff816279c9>] ? schedule_timeout+0x189/0x2e0
[  360.099128]  [<ffffffff812c0ab4>] _xfs_log_force+0x64/0x240
[  360.115616]  [<ffffffff812c0e13>] xfs_log_force+0x13/0x40
[  360.132031]  [<ffffffff812c5646>] xfsaild+0x106/0x6d0
[  360.148296]  [<ffffffff812c5540>] ? xfs_trans_ail_update_bulk+0x260/0x260
[  360.164963]  [<ffffffff8105d8d6>] kthread+0x96/0xa0
[  360.181658]  [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
[  360.198516]  [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
[  360.215534]  [<ffffffff8162c090>] ? gs_change+0xb/0xb
[  480.048246] INFO: task ceph-osd:3282 blocked for more than 120 seconds.
[  480.065939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  480.083979] ceph-osd        D ffffffff8180e9c0     0  3282      1 
0x00000004
[  480.102327]  ffff880dbbb9d7b8 0000000000000086 ffff880dbbba0000 
0000000000012200
[  480.121333]  ffff880dbbb9dfd8 ffff880dbbb9c010 0000000000012200 
0000000000012200
[  480.140374]  ffff880dbbb9dfd8 0000000000012200 ffff881038e69810 
ffff880dbbba0000
[  480.159738] Call Trace:
[  480.178812]  [<ffffffff816296e4>] schedule+0x24/0x70
[  480.198230]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[  480.217775]  [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
[  480.237500]  [<ffffffff8162888a>] __down+0x6a/0xb0
[  480.257254]  [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
[  480.277179]  [<ffffffff8106310b>] down+0x3b/0x50
[  480.297102]  [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
[  480.317000]  [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
[  480.337193]  [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
[  480.357648]  [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
[  480.378666]  [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
[  480.399694]  [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
[  480.421093]  [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
[  480.442728]  [<ffffffff81282e67>] ? kmem_zone_alloc+0x77/0xf0
[  480.464717]  [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
[  480.486433]  [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
[  480.508220]  [<ffffffff8129350f>] ? xfs_bmap_del_extent+0x45f/0xac0
[  480.530335]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[  480.552300]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[  480.574439]  [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
[  480.596523]  [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
[  480.618126]  [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
[  480.639105]  [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
[  480.659636]  [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
[  480.659645]  [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
[  480.659647]  [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
[  480.659650]  [<ffffffff8112481a>] notify_change+0x13a/0x2e0
[  480.659652]  [<ffffffff8110a28f>] do_truncate+0x5f/0x90
[  480.659654]  [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
[  480.659657]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[  480.659659] INFO: task ceph-osd:3283 blocked for more than 120 seconds.
[  480.659659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  480.659661] ceph-osd        D 0000000000000001     0  3283      1 
0x00000004
[  480.659663]  ffff880dbbb9f678 0000000000000086 ffff880dbbba1810 
0000000000012200
[  480.659666]  ffff880dbbb9ffd8 ffff880dbbb9e010 0000000000012200 
0000000000012200
[  480.659669]  ffff880dbbb9ffd8 0000000000012200 ffff881036ab1810 
ffff880dbbba1810
[  480.659671] Call Trace:
[  480.659674]  [<ffffffff816296e4>] schedule+0x24/0x70
[  480.659676]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[  480.659679]  [<ffffffff8106aa44>] ? T.1708+0x44/0x50
[  480.659681]  [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
[  480.659684]  [<ffffffff81628d54>] wait_for_common+0xc4/0x160
[  480.659686]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[  480.659688]  [<ffffffff81628e98>] wait_for_completion+0x18/0x20
[  480.659691]  [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
[  480.659693]  [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
[  480.659695]  [<ffffffff81066215>] ? check_preempt_curr+0x75/0xa0
[  480.659698]  [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
[  480.659700]  [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
[  480.659702]  [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
[  480.659704]  [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
[  480.659707]  [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
[  480.659709]  [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
[  480.659711]  [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
[  480.659713]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[  480.659717]  [<ffffffff810cc4a2>] __writepage+0x12/0x40
[  480.659720]  [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
[  480.659722]  [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
[  480.659724]  [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[  480.659726]  [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
[  480.659728]  [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
[  480.659731]  [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
[  480.659733]  [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[  480.659735]  [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[  480.659737]  [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[  480.659739]  [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
[  480.659742]  [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
[  480.659745]  [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
[  480.659747]  [<ffffffff81109240>] ? filp_close+0x60/0x90
[  480.659749]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[  480.659756] INFO: task flush-8:64:3184 blocked for more than 120 seconds.
[  480.659757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  480.659758] flush-8:64      D ffffffff8180e9c0     0  3184      2 
0x00000000
[  480.659761]  ffff880dbcf31920 0000000000000046 ffff880dbdbcc830 
0000000000012200
[  480.659763]  ffff880dbcf31fd8 ffff880dbcf30010 0000000000012200 
0000000000012200
[  480.659766]  ffff880dbcf31fd8 0000000000012200 ffff881038e89810 
ffff880dbdbcc830
[  480.659768] Call Trace:
[  480.659772]  [<ffffffff810c34e0>] ? __lock_page+0x70/0x70
[  480.659774]  [<ffffffff816296e4>] schedule+0x24/0x70
[  480.659777]  [<ffffffff816297b7>] io_schedule+0x87/0xd0
[  480.659779]  [<ffffffff810c34e9>] sleep_on_page+0x9/0x10
[  480.659781]  [<ffffffff81627bd2>] __wait_on_bit_lock+0x52/0xb0
[  480.659783]  [<ffffffff810c547d>] ? find_get_pages_tag+0xcd/0x180
[  480.659785]  [<ffffffff810c34d2>] __lock_page+0x62/0x70
[  480.659789]  [<ffffffff8105de10>] ? autoremove_wake_function+0x40/0x40
[  480.659791]  [<ffffffff810cf190>] ? pagevec_lookup_tag+0x20/0x30
[  480.659793]  [<ffffffff810cd736>] write_cache_pages+0x376/0x4c0
[  480.659796]  [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[  480.659798]  [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[  480.659800]  [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[  480.659802]  [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[  480.659805]  [<ffffffff811308ff>] writeback_single_inode+0x18f/0x400
[  480.659807]  [<ffffffff81130d06>] writeback_sb_inodes+0x196/0x260
[  480.659809]  [<ffffffff81130e66>] __writeback_inodes_wb+0x96/0xc0
[  480.659811]  [<ffffffff811317bb>] wb_writeback+0x1eb/0x2c0
[  480.659814]  [<ffffffff8104c1a1>] ? try_to_del_timer_sync+0x81/0xe0
[  480.659816]  [<ffffffff81131a89>] wb_do_writeback+0x1f9/0x250
[  480.659818]  [<ffffffff81131b7a>] bdi_writeback_thread+0x9a/0x230
[  480.659820]  [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
[  480.659822]  [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
[  480.659824]  [<ffffffff8105d8d6>] kthread+0x96/0xa0
[  480.659827]  [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
[  480.659829]  [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
[  480.659832]  [<ffffffff8162c090>] ? gs_change+0xb/0xb

Am 26.06.2012 19:49, schrieb Stefan Priebe:
> I would but both XFS and btrfs are crashing after a short period.
>
> XFS crashes with this one:
> [  479.732636] INFO: task ceph-osd:3217 blocked for more than 120 seconds.
> [  479.747724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  479.763534] ceph-osd        D ffffffff8180e9c0     0  3217      1
> 0x00000000
> [  479.779837]  ffff880bc4321bd8 0000000000000082 ffff880bc5694830
> 0000000000012200
> [  479.779840]  ffff880bc4321fd8 ffff880bc4320010 0000000000012200
> 0000000000012200
> [  479.779841]  ffff880bc4321fd8 0000000000012200 ffff880e40ea9810
> ffff880bc5694830
> [  479.779843] Call Trace:
> [  479.779850]  [<ffffffff816296e4>] schedule+0x24/0x70
> [  479.779853]  [<ffffffff812c2049>] xlog_wait+0x69/0x90
> [  479.779856]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
> [  479.779858]  [<ffffffff812c23b3>] xlog_cil_push+0x343/0x3c0
> [  479.779861]  [<ffffffff8126ce09>] ? xfs_buf_unlock+0x19/0x70
> [  479.779862]  [<ffffffff812c2ab1>] xlog_cil_force_lsn+0x101/0x110
> [  479.779864]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
> [  479.779865]  [<ffffffff812bcd77>] ? xfs_trans_free_items+0x87/0xb0
> [  479.779867]  [<ffffffff812c07c8>] _xfs_log_force_lsn+0x48/0x290
> [  479.779871]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
> [  479.779872]  [<ffffffff812bdfdb>] xfs_trans_commit+0x24b/0x260
> [  479.779875]  [<ffffffff81271e9d>] xfs_fs_log_dummy+0x5d/0x90
> [  479.779877]  [<ffffffff812bed9c>] ? xfs_log_need_covered+0x7c/0xc0
> [  479.779879]  [<ffffffff8127d378>] xfs_quiesce_data+0x88/0x90
> [  479.779881]  [<ffffffff8127b428>] xfs_fs_sync_fs+0x28/0x60
> [  479.779884]  [<ffffffff811363ae>] __sync_filesystem+0x5e/0x90
> [  479.779885]  [<ffffffff811364b3>] sync_filesystem+0x43/0x60
> [  479.779887]  [<ffffffff81136518>] sys_syncfs+0x48/0x80
> [  479.779890]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
>
>
> Am 26.06.2012 18:59, schrieb Mark Nelson:
>> On 06/26/2012 11:43 AM, Stefan Priebe wrote:
>>> Am 26.06.2012 18:29, schrieb Mark Nelson:
>>>> On 06/26/2012 11:15 AM, Stefan Priebe wrote:
>>>> Hi Stefan,
>>>>
>>>> If you can, it would be really interesting to see the blktrace results
>>>> during these tests for both xfs and btrfs. blktrace is in the ubuntu
>>>> repositories and can be run quite easily from the command line during
>>>> your test.
>>>
>>> Sure any special parameters? I have 4 SSDs per OSD Server.
>>>
>>> Or just blktrace -o file?
>>>
>>> Stefan
>>
>> For each device you run it on you'll get one file per core.  There may
>> be some performance impact if you run blktrace on every device per node.
>>   If your data is well distributed, even a trace for one OSD (per test)
>> would be interesting.
>>
>> so blktrace -o <outfile prefix> -d <device> where device is your first
>> OSD or something.  If you can do it for both btrfs and xfs and maybe run
>> each test for a couple of minutes that might be enough.
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FS / Kernel question choosing the correct kernel version

[Stefan Priebe]

Or these:
  359.933112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  359.949567] ceph-osd        D ffffffff8180e9c0     0  3841      1 
0x00000000
[  359.966452]  ffff880db567d7b8 0000000000000082 ffff880db6c30000 
0000000000012200
[  359.983845]  ffff880db567dfd8 ffff880db567c010 0000000000012200 
0000000000012200
[  360.001308]  ffff880db567dfd8 0000000000012200 ffff881038ee1810 
ffff880db6c30000
[  360.019065] Call Trace:
[  360.036550]  [<ffffffff816296e4>] schedule+0x24/0x70
[  360.054562]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[  360.054566]  [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
[  360.054568]  [<ffffffff8162888a>] __down+0x6a/0xb0
[  360.054571]  [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
[  360.054573]  [<ffffffff8106310b>] down+0x3b/0x50
[  360.054575]  [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
[  360.054576]  [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
[  360.054578]  [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
[  360.054580]  [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
[  360.054581]  [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
[  360.054584]  [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
[  360.054588]  [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
[  360.054589]  [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
[  360.054592]  [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
[  360.054594]  [<ffffffff816296e4>] ? schedule+0x24/0x70
[  360.054596]  [<ffffffff8162a21d>] ? rwsem_down_failed_common+0xbd/0x150
[  360.054599]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
[  360.054601]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[  360.054603]  [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
[  360.054605]  [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
[  360.054607]  [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
[  360.054609]  [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
[  360.054611]  [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
[  360.054613]  [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
[  360.054615]  [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
[  360.054617]  [<ffffffff8112481a>] notify_change+0x13a/0x2e0
[  360.054619]  [<ffffffff8110a28f>] do_truncate+0x5f/0x90
[  360.054621]  [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
[  360.054623]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
[  360.054625] INFO: task ceph-osd:3843 blocked for more than 120 seconds.
[  360.054626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  360.054627] ceph-osd        D 0000000000000002     0  3843      1 
0x00000000
[  360.054629]  ffff880db5689678 0000000000000082 ffff880db5681810 
0000000000012200
[  360.054631]  ffff880db5689fd8 ffff880db5688010 0000000000012200 
0000000000012200
[  360.054633]  ffff880db5689fd8 0000000000012200 ffff881038f3e040 
ffff880db5681810
[  360.054635] Call Trace:
[  360.054637]  [<ffffffff816296e4>] schedule+0x24/0x70
[  360.054639]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
[  360.054641]  [<ffffffff8106aa44>] ? T.1708+0x44/0x50
[  360.054643]  [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
[  360.054645]  [<ffffffff81628d54>] wait_for_common+0xc4/0x160
[  360.054647]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
[  360.054649]  [<ffffffff81628e98>] wait_for_completion+0x18/0x20
[  360.054650]  [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
[  360.054652]  [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
[  360.054654]  [<ffffffff81298301>] ? 
xfs_bmap_add_extent_delay_real+0xff1/0x17a0
[  360.054656]  [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
[  360.054658]  [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
[  360.054659]  [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
[  360.054661]  [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
[  360.054664]  [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
[  360.054666]  [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
[  360.054667]  [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
[  360.054669]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
[  360.054672]  [<ffffffff810cc4a2>] __writepage+0x12/0x40
[  360.054674]  [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
[  360.054675]  [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
[  360.054677]  [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
[  360.054679]  [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
[  360.054680]  [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
[  360.054682]  [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
[  360.054684]  [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
[  360.054686]  [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
[  360.054688]  [<ffffffff810cd90c>] do_writepages+0x1c/0x40
[  360.054689]  [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
[  360.054691]  [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
[  360.054694]  [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
[  360.054696]  [<ffffffff81109240>] ? filp_close+0x60/0x90
[  360.054698]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b

Am 26.06.2012 19:49, schrieb Stefan Priebe:
> and these:
>    359.919732] INFO: task xfsaild/sdd1:2247 blocked for more than 120
> seconds.
> [  359.934620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  359.950059] xfsaild/sdd1    D 0000000000000007     0  2247      2
> 0x00000000
> [  359.966026]  ffff8810349d9cf0 0000000000000046 ffff881034809810
> 0000000000012200
> [  359.982249]  ffff8810349d9fd8 ffff8810349d8010 0000000000012200
> 0000000000012200
> [  359.998634]  ffff8810349d9fd8 0000000000012200 ffff881038ef8000
> ffff881034809810
> [  360.015139] Call Trace:
> [  360.031552]  [<ffffffff816296e4>] schedule+0x24/0x70
> [  360.048356]  [<ffffffff812c2a9a>] xlog_cil_force_lsn+0xea/0x110
> [  360.065363]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
> [  360.082348]  [<ffffffff816279c9>] ? schedule_timeout+0x189/0x2e0
> [  360.099128]  [<ffffffff812c0ab4>] _xfs_log_force+0x64/0x240
> [  360.115616]  [<ffffffff812c0e13>] xfs_log_force+0x13/0x40
> [  360.132031]  [<ffffffff812c5646>] xfsaild+0x106/0x6d0
> [  360.148296]  [<ffffffff812c5540>] ?
> xfs_trans_ail_update_bulk+0x260/0x260
> [  360.164963]  [<ffffffff8105d8d6>] kthread+0x96/0xa0
> [  360.181658]  [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
> [  360.198516]  [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
> [  360.215534]  [<ffffffff8162c090>] ? gs_change+0xb/0xb
> [  480.048246] INFO: task ceph-osd:3282 blocked for more than 120 seconds.
> [  480.065939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  480.083979] ceph-osd        D ffffffff8180e9c0     0  3282      1
> 0x00000004
> [  480.102327]  ffff880dbbb9d7b8 0000000000000086 ffff880dbbba0000
> 0000000000012200
> [  480.121333]  ffff880dbbb9dfd8 ffff880dbbb9c010 0000000000012200
> 0000000000012200
> [  480.140374]  ffff880dbbb9dfd8 0000000000012200 ffff881038e69810
> ffff880dbbba0000
> [  480.159738] Call Trace:
> [  480.178812]  [<ffffffff816296e4>] schedule+0x24/0x70
> [  480.198230]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
> [  480.217775]  [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
> [  480.237500]  [<ffffffff8162888a>] __down+0x6a/0xb0
> [  480.257254]  [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
> [  480.277179]  [<ffffffff8106310b>] down+0x3b/0x50
> [  480.297102]  [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
> [  480.317000]  [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
> [  480.337193]  [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
> [  480.357648]  [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
> [  480.378666]  [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
> [  480.399694]  [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
> [  480.421093]  [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
> [  480.442728]  [<ffffffff81282e67>] ? kmem_zone_alloc+0x77/0xf0
> [  480.464717]  [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
> [  480.486433]  [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
> [  480.508220]  [<ffffffff8129350f>] ? xfs_bmap_del_extent+0x45f/0xac0
> [  480.530335]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
> [  480.552300]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
> [  480.574439]  [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
> [  480.596523]  [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
> [  480.618126]  [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
> [  480.639105]  [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
> [  480.659636]  [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
> [  480.659645]  [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
> [  480.659647]  [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
> [  480.659650]  [<ffffffff8112481a>] notify_change+0x13a/0x2e0
> [  480.659652]  [<ffffffff8110a28f>] do_truncate+0x5f/0x90
> [  480.659654]  [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
> [  480.659657]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
> [  480.659659] INFO: task ceph-osd:3283 blocked for more than 120 seconds.
> [  480.659659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  480.659661] ceph-osd        D 0000000000000001     0  3283      1
> 0x00000004
> [  480.659663]  ffff880dbbb9f678 0000000000000086 ffff880dbbba1810
> 0000000000012200
> [  480.659666]  ffff880dbbb9ffd8 ffff880dbbb9e010 0000000000012200
> 0000000000012200
> [  480.659669]  ffff880dbbb9ffd8 0000000000012200 ffff881036ab1810
> ffff880dbbba1810
> [  480.659671] Call Trace:
> [  480.659674]  [<ffffffff816296e4>] schedule+0x24/0x70
> [  480.659676]  [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
> [  480.659679]  [<ffffffff8106aa44>] ? T.1708+0x44/0x50
> [  480.659681]  [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
> [  480.659684]  [<ffffffff81628d54>] wait_for_common+0xc4/0x160
> [  480.659686]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
> [  480.659688]  [<ffffffff81628e98>] wait_for_completion+0x18/0x20
> [  480.659691]  [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
> [  480.659693]  [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
> [  480.659695]  [<ffffffff81066215>] ? check_preempt_curr+0x75/0xa0
> [  480.659698]  [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
> [  480.659700]  [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
> [  480.659702]  [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
> [  480.659704]  [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
> [  480.659707]  [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
> [  480.659709]  [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
> [  480.659711]  [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
> [  480.659713]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
> [  480.659717]  [<ffffffff810cc4a2>] __writepage+0x12/0x40
> [  480.659720]  [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
> [  480.659722]  [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
> [  480.659724]  [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
> [  480.659726]  [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
> [  480.659728]  [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
> [  480.659731]  [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
> [  480.659733]  [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
> [  480.659735]  [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
> [  480.659737]  [<ffffffff810cd90c>] do_writepages+0x1c/0x40
> [  480.659739]  [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
> [  480.659742]  [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
> [  480.659745]  [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
> [  480.659747]  [<ffffffff81109240>] ? filp_close+0x60/0x90
> [  480.659749]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
> [  480.659756] INFO: task flush-8:64:3184 blocked for more than 120
> seconds.
> [  480.659757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  480.659758] flush-8:64      D ffffffff8180e9c0     0  3184      2
> 0x00000000
> [  480.659761]  ffff880dbcf31920 0000000000000046 ffff880dbdbcc830
> 0000000000012200
> [  480.659763]  ffff880dbcf31fd8 ffff880dbcf30010 0000000000012200
> 0000000000012200
> [  480.659766]  ffff880dbcf31fd8 0000000000012200 ffff881038e89810
> ffff880dbdbcc830
> [  480.659768] Call Trace:
> [  480.659772]  [<ffffffff810c34e0>] ? __lock_page+0x70/0x70
> [  480.659774]  [<ffffffff816296e4>] schedule+0x24/0x70
> [  480.659777]  [<ffffffff816297b7>] io_schedule+0x87/0xd0
> [  480.659779]  [<ffffffff810c34e9>] sleep_on_page+0x9/0x10
> [  480.659781]  [<ffffffff81627bd2>] __wait_on_bit_lock+0x52/0xb0
> [  480.659783]  [<ffffffff810c547d>] ? find_get_pages_tag+0xcd/0x180
> [  480.659785]  [<ffffffff810c34d2>] __lock_page+0x62/0x70
> [  480.659789]  [<ffffffff8105de10>] ? autoremove_wake_function+0x40/0x40
> [  480.659791]  [<ffffffff810cf190>] ? pagevec_lookup_tag+0x20/0x30
> [  480.659793]  [<ffffffff810cd736>] write_cache_pages+0x376/0x4c0
> [  480.659796]  [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
> [  480.659798]  [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
> [  480.659800]  [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
> [  480.659802]  [<ffffffff810cd90c>] do_writepages+0x1c/0x40
> [  480.659805]  [<ffffffff811308ff>] writeback_single_inode+0x18f/0x400
> [  480.659807]  [<ffffffff81130d06>] writeback_sb_inodes+0x196/0x260
> [  480.659809]  [<ffffffff81130e66>] __writeback_inodes_wb+0x96/0xc0
> [  480.659811]  [<ffffffff811317bb>] wb_writeback+0x1eb/0x2c0
> [  480.659814]  [<ffffffff8104c1a1>] ? try_to_del_timer_sync+0x81/0xe0
> [  480.659816]  [<ffffffff81131a89>] wb_do_writeback+0x1f9/0x250
> [  480.659818]  [<ffffffff81131b7a>] bdi_writeback_thread+0x9a/0x230
> [  480.659820]  [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
> [  480.659822]  [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
> [  480.659824]  [<ffffffff8105d8d6>] kthread+0x96/0xa0
> [  480.659827]  [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
> [  480.659829]  [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
> [  480.659832]  [<ffffffff8162c090>] ? gs_change+0xb/0xb
>
> Am 26.06.2012 19:49, schrieb Stefan Priebe:
>> I would but both XFS and btrfs are crashing after a short period.
>>
>> XFS crashes with this one:
>> [  479.732636] INFO: task ceph-osd:3217 blocked for more than 120
>> seconds.
>> [  479.747724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [  479.763534] ceph-osd        D ffffffff8180e9c0     0  3217      1
>> 0x00000000
>> [  479.779837]  ffff880bc4321bd8 0000000000000082 ffff880bc5694830
>> 0000000000012200
>> [  479.779840]  ffff880bc4321fd8 ffff880bc4320010 0000000000012200
>> 0000000000012200
>> [  479.779841]  ffff880bc4321fd8 0000000000012200 ffff880e40ea9810
>> ffff880bc5694830
>> [  479.779843] Call Trace:
>> [  479.779850]  [<ffffffff816296e4>] schedule+0x24/0x70
>> [  479.779853]  [<ffffffff812c2049>] xlog_wait+0x69/0x90
>> [  479.779856]  [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
>> [  479.779858]  [<ffffffff812c23b3>] xlog_cil_push+0x343/0x3c0
>> [  479.779861]  [<ffffffff8126ce09>] ? xfs_buf_unlock+0x19/0x70
>> [  479.779862]  [<ffffffff812c2ab1>] xlog_cil_force_lsn+0x101/0x110
>> [  479.779864]  [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
>> [  479.779865]  [<ffffffff812bcd77>] ? xfs_trans_free_items+0x87/0xb0
>> [  479.779867]  [<ffffffff812c07c8>] _xfs_log_force_lsn+0x48/0x290
>> [  479.779871]  [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
>> [  479.779872]  [<ffffffff812bdfdb>] xfs_trans_commit+0x24b/0x260
>> [  479.779875]  [<ffffffff81271e9d>] xfs_fs_log_dummy+0x5d/0x90
>> [  479.779877]  [<ffffffff812bed9c>] ? xfs_log_need_covered+0x7c/0xc0
>> [  479.779879]  [<ffffffff8127d378>] xfs_quiesce_data+0x88/0x90
>> [  479.779881]  [<ffffffff8127b428>] xfs_fs_sync_fs+0x28/0x60
>> [  479.779884]  [<ffffffff811363ae>] __sync_filesystem+0x5e/0x90
>> [  479.779885]  [<ffffffff811364b3>] sync_filesystem+0x43/0x60
>> [  479.779887]  [<ffffffff81136518>] sys_syncfs+0x48/0x80
>> [  479.779890]  [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
>>
>>
>> Am 26.06.2012 18:59, schrieb Mark Nelson:
>>> On 06/26/2012 11:43 AM, Stefan Priebe wrote:
>>>> Am 26.06.2012 18:29, schrieb Mark Nelson:
>>>>> On 06/26/2012 11:15 AM, Stefan Priebe wrote:
>>>>> Hi Stefan,
>>>>>
>>>>> If you can, it would be really interesting to see the blktrace results
>>>>> during these tests for both xfs and btrfs. blktrace is in the ubuntu
>>>>> repositories and can be run quite easily from the command line during
>>>>> your test.
>>>>
>>>> Sure any special parameters? I have 4 SSDs per OSD Server.
>>>>
>>>> Or just blktrace -o file?
>>>>
>>>> Stefan
>>>
>>> For each device you run it on you'll get one file per core.  There may
>>> be some performance impact if you run blktrace on every device per node.
>>>   If your data is well distributed, even a trace for one OSD (per test)
>>> would be interesting.
>>>
>>> so blktrace -o <outfile prefix> -d <device> where device is your first
>>> OSD or something.  If you can do it for both btrfs and xfs and maybe run
>>> each test for a couple of minutes that might be enough.
>>>
>>> Mark
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: FS / Kernel question choosing the correct kernel version

[Mark Nelson]

On 06/26/2012 01:04 PM, Stefan Priebe wrote:
> Or these:
> 359.933112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 359.949567] ceph-osd D ffffffff8180e9c0 0 3841 1 0x00000000
> [ 359.966452] ffff880db567d7b8 0000000000000082 ffff880db6c30000
> 0000000000012200
> [ 359.983845] ffff880db567dfd8 ffff880db567c010 0000000000012200
> 0000000000012200
> [ 360.001308] ffff880db567dfd8 0000000000012200 ffff881038ee1810
> ffff880db6c30000
> [ 360.019065] Call Trace:
> [ 360.036550] [<ffffffff816296e4>] schedule+0x24/0x70
> [ 360.054562] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
> [ 360.054566] [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
> [ 360.054568] [<ffffffff8162888a>] __down+0x6a/0xb0
> [ 360.054571] [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
> [ 360.054573] [<ffffffff8106310b>] down+0x3b/0x50
> [ 360.054575] [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
> [ 360.054576] [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
> [ 360.054578] [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
> [ 360.054580] [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
> [ 360.054581] [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
> [ 360.054584] [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
> [ 360.054588] [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
> [ 360.054589] [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
> [ 360.054592] [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
> [ 360.054594] [<ffffffff816296e4>] ? schedule+0x24/0x70
> [ 360.054596] [<ffffffff8162a21d>] ? rwsem_down_failed_common+0xbd/0x150
> [ 360.054599] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
> [ 360.054601] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
> [ 360.054603] [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
> [ 360.054605] [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
> [ 360.054607] [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
> [ 360.054609] [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
> [ 360.054611] [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
> [ 360.054613] [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
> [ 360.054615] [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
> [ 360.054617] [<ffffffff8112481a>] notify_change+0x13a/0x2e0
> [ 360.054619] [<ffffffff8110a28f>] do_truncate+0x5f/0x90
> [ 360.054621] [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
> [ 360.054623] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
> [ 360.054625] INFO: task ceph-osd:3843 blocked for more than 120 seconds.
> [ 360.054626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 360.054627] ceph-osd D 0000000000000002 0 3843 1 0x00000000
> [ 360.054629] ffff880db5689678 0000000000000082 ffff880db5681810
> 0000000000012200
> [ 360.054631] ffff880db5689fd8 ffff880db5688010 0000000000012200
> 0000000000012200
> [ 360.054633] ffff880db5689fd8 0000000000012200 ffff881038f3e040
> ffff880db5681810
> [ 360.054635] Call Trace:
> [ 360.054637] [<ffffffff816296e4>] schedule+0x24/0x70
> [ 360.054639] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
> [ 360.054641] [<ffffffff8106aa44>] ? T.1708+0x44/0x50
> [ 360.054643] [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
> [ 360.054645] [<ffffffff81628d54>] wait_for_common+0xc4/0x160
> [ 360.054647] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
> [ 360.054649] [<ffffffff81628e98>] wait_for_completion+0x18/0x20
> [ 360.054650] [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
> [ 360.054652] [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
> [ 360.054654] [<ffffffff81298301>] ?
> xfs_bmap_add_extent_delay_real+0xff1/0x17a0
> [ 360.054656] [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
> [ 360.054658] [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
> [ 360.054659] [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
> [ 360.054661] [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
> [ 360.054664] [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
> [ 360.054666] [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
> [ 360.054667] [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
> [ 360.054669] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
> [ 360.054672] [<ffffffff810cc4a2>] __writepage+0x12/0x40
> [ 360.054674] [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
> [ 360.054675] [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
> [ 360.054677] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
> [ 360.054679] [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
> [ 360.054680] [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
> [ 360.054682] [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
> [ 360.054684] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
> [ 360.054686] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
> [ 360.054688] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
> [ 360.054689] [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
> [ 360.054691] [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
> [ 360.054694] [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
> [ 360.054696] [<ffffffff81109240>] ? filp_close+0x60/0x90
> [ 360.054698] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
>
> Am 26.06.2012 19:49, schrieb Stefan Priebe:
>> and these:
>> 359.919732] INFO: task xfsaild/sdd1:2247 blocked for more than 120
>> seconds.
>> [ 359.934620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 359.950059] xfsaild/sdd1 D 0000000000000007 0 2247 2
>> 0x00000000
>> [ 359.966026] ffff8810349d9cf0 0000000000000046 ffff881034809810
>> 0000000000012200
>> [ 359.982249] ffff8810349d9fd8 ffff8810349d8010 0000000000012200
>> 0000000000012200
>> [ 359.998634] ffff8810349d9fd8 0000000000012200 ffff881038ef8000
>> ffff881034809810
>> [ 360.015139] Call Trace:
>> [ 360.031552] [<ffffffff816296e4>] schedule+0x24/0x70
>> [ 360.048356] [<ffffffff812c2a9a>] xlog_cil_force_lsn+0xea/0x110
>> [ 360.065363] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
>> [ 360.082348] [<ffffffff816279c9>] ? schedule_timeout+0x189/0x2e0
>> [ 360.099128] [<ffffffff812c0ab4>] _xfs_log_force+0x64/0x240
>> [ 360.115616] [<ffffffff812c0e13>] xfs_log_force+0x13/0x40
>> [ 360.132031] [<ffffffff812c5646>] xfsaild+0x106/0x6d0
>> [ 360.148296] [<ffffffff812c5540>] ?
>> xfs_trans_ail_update_bulk+0x260/0x260
>> [ 360.164963] [<ffffffff8105d8d6>] kthread+0x96/0xa0
>> [ 360.181658] [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
>> [ 360.198516] [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
>> [ 360.215534] [<ffffffff8162c090>] ? gs_change+0xb/0xb
>> [ 480.048246] INFO: task ceph-osd:3282 blocked for more than 120 seconds.
>> [ 480.065939] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 480.083979] ceph-osd D ffffffff8180e9c0 0 3282 1
>> 0x00000004
>> [ 480.102327] ffff880dbbb9d7b8 0000000000000086 ffff880dbbba0000
>> 0000000000012200
>> [ 480.121333] ffff880dbbb9dfd8 ffff880dbbb9c010 0000000000012200
>> 0000000000012200
>> [ 480.140374] ffff880dbbb9dfd8 0000000000012200 ffff881038e69810
>> ffff880dbbba0000
>> [ 480.159738] Call Trace:
>> [ 480.178812] [<ffffffff816296e4>] schedule+0x24/0x70
>> [ 480.198230] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
>> [ 480.217775] [<ffffffff812affe4>] ? xfs_iext_bno_to_ext+0x84/0x160
>> [ 480.237500] [<ffffffff8162888a>] __down+0x6a/0xb0
>> [ 480.257254] [<ffffffff8126d885>] ? _xfs_buf_find+0xe5/0x230
>> [ 480.277179] [<ffffffff8106310b>] down+0x3b/0x50
>> [ 480.297102] [<ffffffff8126d640>] xfs_buf_lock+0x40/0xe0
>> [ 480.317000] [<ffffffff8126d885>] _xfs_buf_find+0xe5/0x230
>> [ 480.337193] [<ffffffff8126d9ff>] xfs_buf_get+0x2f/0x1b0
>> [ 480.357648] [<ffffffff812bb2a2>] ? xfs_icsb_modify_counters+0xa2/0x190
>> [ 480.378666] [<ffffffff8126e1f7>] xfs_buf_read+0x27/0x100
>> [ 480.399694] [<ffffffff812c64c7>] xfs_trans_read_buf+0x1f7/0x430
>> [ 480.421093] [<ffffffff81283ab1>] xfs_read_agf+0x61/0x190
>> [ 480.442728] [<ffffffff81282e67>] ? kmem_zone_alloc+0x77/0xf0
>> [ 480.464717] [<ffffffff81283c14>] xfs_alloc_read_agf+0x34/0xd0
>> [ 480.486433] [<ffffffff81286fd7>] xfs_alloc_fix_freelist+0x427/0x4a0
>> [ 480.508220] [<ffffffff8129350f>] ? xfs_bmap_del_extent+0x45f/0xac0
>> [ 480.530335] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
>> [ 480.552300] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
>> [ 480.574439] [<ffffffff812bab63>] ? xfs_perag_get+0x33/0xb0
>> [ 480.596523] [<ffffffff812870e7>] ? xfs_free_extent+0x97/0x140
>> [ 480.618126] [<ffffffff812870fb>] xfs_free_extent+0xab/0x140
>> [ 480.639105] [<ffffffff812915bd>] xfs_bmap_finish+0x15d/0x1a0
>> [ 480.659636] [<ffffffff812b3487>] xfs_itruncate_extents+0xf7/0x2a0
>> [ 480.659645] [<ffffffff81278d34>] xfs_setattr_size+0x394/0x410
>> [ 480.659647] [<ffffffff81278de2>] xfs_vn_setattr+0x32/0x40
>> [ 480.659650] [<ffffffff8112481a>] notify_change+0x13a/0x2e0
>> [ 480.659652] [<ffffffff8110a28f>] do_truncate+0x5f/0x90
>> [ 480.659654] [<ffffffff8110a4bf>] sys_truncate+0x12f/0x150
>> [ 480.659657] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
>> [ 480.659659] INFO: task ceph-osd:3283 blocked for more than 120 seconds.
>> [ 480.659659] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 480.659661] ceph-osd D 0000000000000001 0 3283 1
>> 0x00000004
>> [ 480.659663] ffff880dbbb9f678 0000000000000086 ffff880dbbba1810
>> 0000000000012200
>> [ 480.659666] ffff880dbbb9ffd8 ffff880dbbb9e010 0000000000012200
>> 0000000000012200
>> [ 480.659669] ffff880dbbb9ffd8 0000000000012200 ffff881036ab1810
>> ffff880dbbba1810
>> [ 480.659671] Call Trace:
>> [ 480.659674] [<ffffffff816296e4>] schedule+0x24/0x70
>> [ 480.659676] [<ffffffff81627a3d>] schedule_timeout+0x1fd/0x2e0
>> [ 480.659679] [<ffffffff8106aa44>] ? T.1708+0x44/0x50
>> [ 480.659681] [<ffffffff8106ddaf>] ? try_to_wake_up+0x23f/0x2b0
>> [ 480.659684] [<ffffffff81628d54>] wait_for_common+0xc4/0x160
>> [ 480.659686] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
>> [ 480.659688] [<ffffffff81628e98>] wait_for_completion+0x18/0x20
>> [ 480.659691] [<ffffffff81283a3d>] xfs_alloc_vextent+0x6d/0x80
>> [ 480.659693] [<ffffffff812922b1>] xfs_bmap_btalloc+0x2a1/0x7d0
>> [ 480.659695] [<ffffffff81066215>] ? check_preempt_curr+0x75/0xa0
>> [ 480.659698] [<ffffffff81287780>] ? __xfs_alloc_vextent+0x5f0/0x5f0
>> [ 480.659700] [<ffffffff812927e9>] xfs_bmap_alloc+0x9/0x10
>> [ 480.659702] [<ffffffff81298b78>] xfs_bmapi_allocate+0xc8/0x2c0
>> [ 480.659704] [<ffffffff81299279>] xfs_bmapi_write+0x509/0x780
>> [ 480.659707] [<ffffffff81276ab4>] xfs_iomap_write_allocate+0x134/0x370
>> [ 480.659709] [<ffffffff8126b325>] xfs_map_blocks+0x155/0x230
>> [ 480.659711] [<ffffffff8126b597>] xfs_vm_writepage+0x197/0x540
>> [ 480.659713] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
>> [ 480.659717] [<ffffffff810cc4a2>] __writepage+0x12/0x40
>> [ 480.659720] [<ffffffff810cd5f6>] write_cache_pages+0x236/0x4c0
>> [ 480.659722] [<ffffffff812c27ef>] ? xfs_log_commit_cil+0x3bf/0x480
>> [ 480.659724] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
>> [ 480.659726] [<ffffffff812bccac>] ? xfs_trans_free+0x5c/0x70
>> [ 480.659728] [<ffffffff812bde3b>] ? xfs_trans_commit+0xab/0x260
>> [ 480.659731] [<ffffffff8127362e>] ? xfs_iunlock+0x7e/0xd0
>> [ 480.659733] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
>> [ 480.659735] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
>> [ 480.659737] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
>> [ 480.659739] [<ffffffff810c3b7b>] __filemap_fdatawrite_range+0x5b/0x60
>> [ 480.659742] [<ffffffff810c3e1e>] filemap_fdatawrite_range+0xe/0x10
>> [ 480.659745] [<ffffffff811361ae>] sys_sync_file_range+0x14e/0x180
>> [ 480.659747] [<ffffffff81109240>] ? filp_close+0x60/0x90
>> [ 480.659749] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b
>> [ 480.659756] INFO: task flush-8:64:3184 blocked for more than 120
>> seconds.
>> [ 480.659757] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 480.659758] flush-8:64 D ffffffff8180e9c0 0 3184 2
>> 0x00000000
>> [ 480.659761] ffff880dbcf31920 0000000000000046 ffff880dbdbcc830
>> 0000000000012200
>> [ 480.659763] ffff880dbcf31fd8 ffff880dbcf30010 0000000000012200
>> 0000000000012200
>> [ 480.659766] ffff880dbcf31fd8 0000000000012200 ffff881038e89810
>> ffff880dbdbcc830
>> [ 480.659768] Call Trace:
>> [ 480.659772] [<ffffffff810c34e0>] ? __lock_page+0x70/0x70
>> [ 480.659774] [<ffffffff816296e4>] schedule+0x24/0x70
>> [ 480.659777] [<ffffffff816297b7>] io_schedule+0x87/0xd0
>> [ 480.659779] [<ffffffff810c34e9>] sleep_on_page+0x9/0x10
>> [ 480.659781] [<ffffffff81627bd2>] __wait_on_bit_lock+0x52/0xb0
>> [ 480.659783] [<ffffffff810c547d>] ? find_get_pages_tag+0xcd/0x180
>> [ 480.659785] [<ffffffff810c34d2>] __lock_page+0x62/0x70
>> [ 480.659789] [<ffffffff8105de10>] ? autoremove_wake_function+0x40/0x40
>> [ 480.659791] [<ffffffff810cf190>] ? pagevec_lookup_tag+0x20/0x30
>> [ 480.659793] [<ffffffff810cd736>] write_cache_pages+0x376/0x4c0
>> [ 480.659796] [<ffffffff810cc490>] ? set_page_dirty+0x70/0x70
>> [ 480.659798] [<ffffffff810cd8cc>] generic_writepages+0x4c/0x70
>> [ 480.659800] [<ffffffff81269fef>] xfs_vm_writepages+0x4f/0x60
>> [ 480.659802] [<ffffffff810cd90c>] do_writepages+0x1c/0x40
>> [ 480.659805] [<ffffffff811308ff>] writeback_single_inode+0x18f/0x400
>> [ 480.659807] [<ffffffff81130d06>] writeback_sb_inodes+0x196/0x260
>> [ 480.659809] [<ffffffff81130e66>] __writeback_inodes_wb+0x96/0xc0
>> [ 480.659811] [<ffffffff811317bb>] wb_writeback+0x1eb/0x2c0
>> [ 480.659814] [<ffffffff8104c1a1>] ? try_to_del_timer_sync+0x81/0xe0
>> [ 480.659816] [<ffffffff81131a89>] wb_do_writeback+0x1f9/0x250
>> [ 480.659818] [<ffffffff81131b7a>] bdi_writeback_thread+0x9a/0x230
>> [ 480.659820] [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
>> [ 480.659822] [<ffffffff81131ae0>] ? wb_do_writeback+0x250/0x250
>> [ 480.659824] [<ffffffff8105d8d6>] kthread+0x96/0xa0
>> [ 480.659827] [<ffffffff8162c094>] kernel_thread_helper+0x4/0x10
>> [ 480.659829] [<ffffffff8105d840>] ? kthread_worker_fn+0x130/0x130
>> [ 480.659832] [<ffffffff8162c090>] ? gs_change+0xb/0xb
>>
>> Am 26.06.2012 19:49, schrieb Stefan Priebe:
>>> I would but both XFS and btrfs are crashing after a short period.
>>>
>>> XFS crashes with this one:
>>> [ 479.732636] INFO: task ceph-osd:3217 blocked for more than 120
>>> seconds.
>>> [ 479.747724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 479.763534] ceph-osd D ffffffff8180e9c0 0 3217 1
>>> 0x00000000
>>> [ 479.779837] ffff880bc4321bd8 0000000000000082 ffff880bc5694830
>>> 0000000000012200
>>> [ 479.779840] ffff880bc4321fd8 ffff880bc4320010 0000000000012200
>>> 0000000000012200
>>> [ 479.779841] ffff880bc4321fd8 0000000000012200 ffff880e40ea9810
>>> ffff880bc5694830
>>> [ 479.779843] Call Trace:
>>> [ 479.779850] [<ffffffff816296e4>] schedule+0x24/0x70
>>> [ 479.779853] [<ffffffff812c2049>] xlog_wait+0x69/0x90
>>> [ 479.779856] [<ffffffff8106de20>] ? try_to_wake_up+0x2b0/0x2b0
>>> [ 479.779858] [<ffffffff812c23b3>] xlog_cil_push+0x343/0x3c0
>>> [ 479.779861] [<ffffffff8126ce09>] ? xfs_buf_unlock+0x19/0x70
>>> [ 479.779862] [<ffffffff812c2ab1>] xlog_cil_force_lsn+0x101/0x110
>>> [ 479.779864] [<ffffffff812bccee>] ? xfs_trans_free_item_desc+0x2e/0x30
>>> [ 479.779865] [<ffffffff812bcd77>] ? xfs_trans_free_items+0x87/0xb0
>>> [ 479.779867] [<ffffffff812c07c8>] _xfs_log_force_lsn+0x48/0x290
>>> [ 479.779871] [<ffffffff8110351b>] ? kmem_cache_free+0x1b/0xf0
>>> [ 479.779872] [<ffffffff812bdfdb>] xfs_trans_commit+0x24b/0x260
>>> [ 479.779875] [<ffffffff81271e9d>] xfs_fs_log_dummy+0x5d/0x90
>>> [ 479.779877] [<ffffffff812bed9c>] ? xfs_log_need_covered+0x7c/0xc0
>>> [ 479.779879] [<ffffffff8127d378>] xfs_quiesce_data+0x88/0x90
>>> [ 479.779881] [<ffffffff8127b428>] xfs_fs_sync_fs+0x28/0x60
>>> [ 479.779884] [<ffffffff811363ae>] __sync_filesystem+0x5e/0x90
>>> [ 479.779885] [<ffffffff811364b3>] sync_filesystem+0x43/0x60
>>> [ 479.779887] [<ffffffff81136518>] sys_syncfs+0x48/0x80
>>> [ 479.779890] [<ffffffff8162ae62>] system_call_fastpath+0x16/0x1b

I'm not really familiar enough with XFS to help here.  :/  Did Ben 
Myer's suggestion to look at /proc/sysrq-trigger back in May provide any 
useful information?

Despite my ignorance here, it kind of seems like most of this is log or 
maybe an AG locking issue?  If you change around the size of the log 
(maybe internal vs external too?) and/or reduce the number of AGs to 1 
does it have any effect?

Mark