[Cluster-devel] GFS2 hangs on running bonnie

[Cluster-devel] GFS2 hangs on running bonnie

[Goldwyn Rodrigues]

Hi,

I am trying to run bonnie on GFS2 but it hangs on every instance of its run.
The kernel is 2.6.35.3 on an i686x2 processor.
gfs2 is formatted with lock_nolock and mounted locally.

The backtrace of the process is -

[20937.467142] bonnie++      S 0000130a     0 22275  11549 0x00000080
[20937.467142]  c14f7cc0 00000082 dfc7eaed 0000130a c0ada140 c0ada140
c0ada140 c0ada140
[20937.467142]  f37c9bc8 c0ada140 c0ada140 00001e2b 00000000 00000000
0000130a f37c9940
[20937.467142]  00000286 c14f7ccc 013ae755 00000000 c14f7d0c c07f928b
00000000 f6c802b8
[20937.467142] Call Trace:
[20937.467142]  [<c07f928b>] schedule_timeout+0x7f/0xad
[20937.467142]  [<c0440aa1>] ? process_timeout+0x0/0xa
[20937.467142]  [<c07f887d>] io_schedule_timeout+0x65/0xa6
[20937.467142]  [<c049c23d>] balance_dirty_pages_ratelimited_nr+0x24d/0x2d2
[20937.467142]  [<f7ed95d0>] ? gfs2_glock_put+0xc5/0xe2 [gfs2]
[20937.467142]  [<c04963b4>] generic_file_buffered_write+0x16d/0x1ba
[20937.467142]  [<c04973bc>] __generic_file_aio_write+0x250/0x281
[20937.467142]  [<c0428d05>] ? update_curr+0x145/0x1fb
[20937.467142]  [<c0497444>] generic_file_aio_write+0x57/0x9e
[20937.467142]  [<f7ee22f5>] gfs2_file_aio_write+0x53/0x5d [gfs2]
[20937.467142]  [<c041842d>] ? lapic_next_event+0x16/0x1a
[20937.467142]  [<c0440694>] ? run_timer_softirq+0x32/0x20d
[20937.467142]  [<c04c5b62>] do_sync_write+0x8a/0xc5
[20937.467142]  [<c05e282c>] ? security_file_permission+0xf/0x11
[20937.467142]  [<c04c5cfa>] ? rw_verify_area+0x98/0xbb
[20937.467142]  [<c04c5ad8>] ? do_sync_write+0x0/0xc5
[20937.467142]  [<c04c6093>] vfs_write+0x80/0xdf
[20937.467142]  [<c04ccc13>] ? path_put+0x15/0x18
[20937.467142]  [<c04c6186>] sys_write+0x3b/0x5d
[20937.467142]  [<c040311f>] sysenter_do_call+0x12/0x28


And the glock workque -

[20937.467142] glock_workque S c7059f4c     0 22238      2 0x00000080
[20937.467142]  c7059f70 00000046 c0447d2a c7059f4c c0ada140 c0ada140 c0ada140 c
0ada140
[20937.467142]  f6786788 c0ada140 c0ada140 cc3d4fb5 00001309 00000000 00001309 f
6786500
[20937.467142]  00000001 c300df00 f6c032f0 c300df04 c7059fbc c0447393 f67867f0 f
6786500
[20937.467142] Call Trace:
[20937.467142]  [<c0447d2a>] ? queue_delayed_work+0xc/0x1e
[20937.467142]  [<c0447393>] worker_thread+0x99/0x1b9
[20937.467142]  [<f7eda167>] ? glock_work_func+0x0/0x105 [gfs2]
[20937.467142]  [<c044a49d>] ? autoremove_wake_function+0x0/0x2f
[20937.467142]  [<c04472fa>] ? worker_thread+0x0/0x1b9
[20937.467142]  [<c044a13f>] kthread+0x5f/0x64
[20937.467142]  [<c044a0e0>] ? kthread+0x0/0x64
[20937.467142]  [<c04036be>] kernel_thread_helper+0x6/0x10
[20937.467142] glock_workque S 000010b2     0 22239      2 0x00000080
[20937.467142]  d72a1f70 00000046 48be3252 000010b2 c0ada140 c0ada140
c0ada140 c0ada140
[20937.467142]  f5d541a8 c0ada140 c0ada140 0000526c 00000000 f676b180
000010b2 f5d53f20
[20937.467142]  00000000 c310df00 f6c032f0 c310df04 d72a1fbc c0447393
f5d54210 f5d53f20
[20937.467142] Call Trace:
[20937.467142]  [<c0447393>] worker_thread+0x99/0x1b9
[20937.467142]  [<f7eda167>] ? glock_work_func+0x0/0x105 [gfs2]
[20937.467142]  [<c044a49d>] ? autoremove_wake_function+0x0/0x2f
[20937.467142]  [<c04472fa>] ? worker_thread+0x0/0x1b9
[20937.467142]  [<c044a13f>] kthread+0x5f/0x64
[20937.467142]  [<c044a0e0>] ? kthread+0x0/0x64
[20937.467142]  [<c04036be>] kernel_thread_helper+0x6/0x10

Let me know if you need any more information.

-- 
Goldwyn

[Cluster-devel] GFS2 hangs on running bonnie

[Steven Whitehouse]

Hi,

On Wed, 2010-08-25 at 18:15 -0500, Goldwyn Rodrigues wrote:
> Hi,
> 
> I am trying to run bonnie on GFS2 but it hangs on every instance of its run.
> The kernel is 2.6.35.3 on an i686x2 processor.
> gfs2 is formatted with lock_nolock and mounted locally.
> 
> The backtrace of the process is -
> 
> [20937.467142] bonnie++      S 0000130a     0 22275  11549 0x00000080
> [20937.467142]  c14f7cc0 00000082 dfc7eaed 0000130a c0ada140 c0ada140
> c0ada140 c0ada140
> [20937.467142]  f37c9bc8 c0ada140 c0ada140 00001e2b 00000000 00000000
> 0000130a f37c9940
> [20937.467142]  00000286 c14f7ccc 013ae755 00000000 c14f7d0c c07f928b
> 00000000 f6c802b8
> [20937.467142] Call Trace:
> [20937.467142]  [<c07f928b>] schedule_timeout+0x7f/0xad
> [20937.467142]  [<c0440aa1>] ? process_timeout+0x0/0xa
> [20937.467142]  [<c07f887d>] io_schedule_timeout+0x65/0xa6
> [20937.467142]  [<c049c23d>] balance_dirty_pages_ratelimited_nr+0x24d/0x2d2

That looks to me as if the page cache has got enough dirty data that the
write has triggered writeback at this point. The question is why that is
apparently stuck. I've not tried bonnie recently, but both postmark and
fsx were working happily on the latest kernels. The trace makes it look
as if it is waiting for I/O to complete,

Steve.