* possible null pointer in xlog_iodone
@ 2013-06-14 17:15 Vlad Bespalov
2013-06-14 19:42 ` Mark Tinguely
2013-06-15 1:03 ` Dave Chinner
0 siblings, 2 replies; 6+ messages in thread
From: Vlad Bespalov @ 2013-06-14 17:15 UTC (permalink / raw)
To: xfs
i`m running an xfs filesystem over device going offline/online and
sometimes offline may be done in parallel with unmounting
at some point i got several crashes with null pointer panic in
xlog_iodone: xlog_t structure taken from input buffer is null
i wonder if the following call path combined with disk online/offline
handling could have led to this crash:
--------------
xfs_unmountfs()
xfs_log_unmount_write(mp)
xlog_state_release_iclog(log)
xlog_sync(log, iclog = log->l_iclog)
(bp=iclog->ic_bp)
xlog_bdstrat(bp)
(iclog->ic_state != XLOG_STATE_ERROR ? )
xfs_buf_iorequest(bp)
xfs_buf_ioend (called with scheduling (*) )
(queues : bp->b_iodone_work,
callback: xlog_iodone)
xfs_log_unmount(mp)
xfs_trans_ail_destroy(mp);
xlog_dealloc_log(mp->m_log); /*frees and nullifies all iclog->ic_log*/
-----------
(after we`ve cleaned up log structures we switch processes*)
xlog_iodone(bp)
{
iclog = bp->private
l = iclog->ic_log
if (XFS_TEST_ERROR((XFS_BUF_GETERROR(bp)), l->l_mp,
XFS_ERRTAG_IODONE_IOERR, XFS_RANDOM_IODONE_IOERR))
{
xfs_buf_ioerror_alert(bp, __func__);
XFS_BUF_STALE(bp);
/*l ?= NULL*/ xfs_force_shutdown(l->l_mp, SHUTDOWN_LOG_IO_ERROR);
}
}
Thanks for your time.
Best regards,
Vlad Bespalov.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: possible null pointer in xlog_iodone
2013-06-14 17:15 possible null pointer in xlog_iodone Vlad Bespalov
@ 2013-06-14 19:42 ` Mark Tinguely
2013-06-15 1:03 ` Dave Chinner
1 sibling, 0 replies; 6+ messages in thread
From: Mark Tinguely @ 2013-06-14 19:42 UTC (permalink / raw)
To: Vlad Bespalov; +Cc: xfs
On 06/14/13 12:15, Vlad Bespalov wrote:
> i`m running an xfs filesystem over device going offline/online and
> sometimes offline may be done in parallel with unmounting
>
> at some point i got several crashes with null pointer panic in
> xlog_iodone: xlog_t structure taken from input buffer is null
>
> i wonder if the following call path combined with disk online/offline
> handling could have led to this crash:
>
> --------------
> xfs_unmountfs()
> xfs_log_unmount_write(mp)
> xlog_state_release_iclog(log)
> xlog_sync(log, iclog = log->l_iclog)
> (bp=iclog->ic_bp)
> xlog_bdstrat(bp)
> (iclog->ic_state != XLOG_STATE_ERROR ? )
> xfs_buf_iorequest(bp)
> xfs_buf_ioend (called with scheduling (*) )
> (queues : bp->b_iodone_work,
> callback: xlog_iodone)
> xfs_log_unmount(mp)
> xfs_trans_ail_destroy(mp);
> xlog_dealloc_log(mp->m_log); /*frees and nullifies all iclog->ic_log*/
> -----------
>
> (after we`ve cleaned up log structures we switch processes*)
> xlog_iodone(bp)
> {
> iclog = bp->private
> l = iclog->ic_log
> if (XFS_TEST_ERROR((XFS_BUF_GETERROR(bp)), l->l_mp,
> XFS_ERRTAG_IODONE_IOERR, XFS_RANDOM_IODONE_IOERR))
> {
>
> xfs_buf_ioerror_alert(bp, __func__);
> XFS_BUF_STALE(bp);
> /*l ?= NULL*/ xfs_force_shutdown(l->l_mp, SHUTDOWN_LOG_IO_ERROR);
> }
> }
>
>
> Thanks for your time.
> Best regards,
> Vlad Bespalov.
>
Hi,
Looks like the log unmount record can't get written because of the disk
being offline.
When the write times out, the log structures are long gone.
I bet if you used memory poisoning, the iclog->ic_log would not work either.
--Mark.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: possible null pointer in xlog_iodone
2013-06-14 17:15 possible null pointer in xlog_iodone Vlad Bespalov
2013-06-14 19:42 ` Mark Tinguely
@ 2013-06-15 1:03 ` Dave Chinner
2013-06-17 14:38 ` Vlad Bespalov
1 sibling, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2013-06-15 1:03 UTC (permalink / raw)
To: Vlad Bespalov; +Cc: xfs
On Fri, Jun 14, 2013 at 09:15:04PM +0400, Vlad Bespalov wrote:
> i`m running an xfs filesystem over device going offline/online and
> sometimes offline may be done in parallel with unmounting
>
> at some point i got several crashes with null pointer panic in
> xlog_iodone: xlog_t structure taken from input buffer is null
>
> i wonder if the following call path combined with disk online/offline
> handling could have led to this crash:
>
> --------------
> xfs_unmountfs()
> xfs_log_unmount_write(mp)
> xlog_state_release_iclog(log)
> xlog_sync(log, iclog = log->l_iclog)
> (bp=iclog->ic_bp)
> xlog_bdstrat(bp)
> (iclog->ic_state != XLOG_STATE_ERROR ? )
> xfs_buf_iorequest(bp)
> xfs_buf_ioend (called with scheduling (*) )
> (queues : bp->b_iodone_work,
> callback: xlog_iodone)
Which is followed by:
if (!(iclog->ic_state == XLOG_STATE_ACTIVE ||
iclog->ic_state == XLOG_STATE_DIRTY)) {
if (!XLOG_FORCED_SHUTDOWN(log)) {
xlog_wait(&iclog->ic_force_wait,
&log->l_icloglock)
Which is supposed to wait for the log IO to complete and hence
xlog_iodone() is supposed to have been run by the time this code
completes.
What kernel are you tesing on? Do you have a script that reproduces
it?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: possible null pointer in xlog_iodone
2013-06-15 1:03 ` Dave Chinner
@ 2013-06-17 14:38 ` Vlad Bespalov
2013-06-18 0:19 ` Dave Chinner
0 siblings, 1 reply; 6+ messages in thread
From: Vlad Bespalov @ 2013-06-17 14:38 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
CentOS 6.4 kernel 2.6.32-358.2.1.el6.x86_64
i`ll try to make a script that can be used out-of-house.
several requests may have failed (un)conveniently, so we have passed
through setting XLOG_IO_ERROR in xfs_log_force_unmount() and
eventually skipped xlog_wait altogether:
-----------------
<1>XFS (<device>): metadata I/O error: block 0xa03e1d ("xlog_iodone")
error 5 buf count 1024
<5>XFS (<device>): xfs_do_force_shutdown(0x2) called from line 1052 of
file fs/xfs/xfs_log.c. Return address = 0xffffffffa0559e51
<1>XFS (<device>): Log I/O Error Detected. Shutting down filesystem
<1>XFS (<device>): Please umount the filesystem and rectify the problem(s)
<4>XFS (<device>): xfs_log_force: error 5 returned.
<5>XFS (<device>): xfs_do_force_shutdown(0x1) called from line 1063 of
file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa0575
<4>XFS (<device>): xfs_log_force: error 5 returned.
<4>XFS (<device>): xfs_log_force: error 5 returned.
...
<1>XFS (mtab~): metadata I/O error: block 0xa03e1b ("xlog_iodone")
error 5 buf count 1024
---------------
Thanks.
On 15 June 2013 05:03, Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Jun 14, 2013 at 09:15:04PM +0400, Vlad Bespalov wrote:
>> i`m running an xfs filesystem over device going offline/online and
>> sometimes offline may be done in parallel with unmounting
>>
>> at some point i got several crashes with null pointer panic in
>> xlog_iodone: xlog_t structure taken from input buffer is null
>>
>> i wonder if the following call path combined with disk online/offline
>> handling could have led to this crash:
>>
>> --------------
>> xfs_unmountfs()
>> xfs_log_unmount_write(mp)
>> xlog_state_release_iclog(log)
>> xlog_sync(log, iclog = log->l_iclog)
>> (bp=iclog->ic_bp)
>> xlog_bdstrat(bp)
>> (iclog->ic_state != XLOG_STATE_ERROR ? )
>> xfs_buf_iorequest(bp)
>> xfs_buf_ioend (called with scheduling (*) )
>> (queues : bp->b_iodone_work,
>> callback: xlog_iodone)
>
> Which is followed by:
>
> if (!(iclog->ic_state == XLOG_STATE_ACTIVE ||
> iclog->ic_state == XLOG_STATE_DIRTY)) {
> if (!XLOG_FORCED_SHUTDOWN(log)) {
> xlog_wait(&iclog->ic_force_wait,
> &log->l_icloglock)
>
> Which is supposed to wait for the log IO to complete and hence
> xlog_iodone() is supposed to have been run by the time this code
> completes.
>
> What kernel are you tesing on? Do you have a script that reproduces
> it?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: possible null pointer in xlog_iodone
2013-06-17 14:38 ` Vlad Bespalov
@ 2013-06-18 0:19 ` Dave Chinner
2013-06-18 0:37 ` Vlad Bespalov
0 siblings, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2013-06-18 0:19 UTC (permalink / raw)
To: Vlad Bespalov; +Cc: xfs
On Mon, Jun 17, 2013 at 06:38:56PM +0400, Vlad Bespalov wrote:
> CentOS 6.4 kernel 2.6.32-358.2.1.el6.x86_64
> i`ll try to make a script that can be used out-of-house.
>
> several requests may have failed (un)conveniently, so we have passed
> through setting XLOG_IO_ERROR in xfs_log_force_unmount() and
> eventually skipped xlog_wait altogether:
> -----------------
> <1>XFS (<device>): metadata I/O error: block 0xa03e1d ("xlog_iodone")
> error 5 buf count 1024
> <5>XFS (<device>): xfs_do_force_shutdown(0x2) called from line 1052 of
> file fs/xfs/xfs_log.c. Return address = 0xffffffffa0559e51
> <1>XFS (<device>): Log I/O Error Detected. Shutting down filesystem
> <1>XFS (<device>): Please umount the filesystem and rectify the problem(s)
> <4>XFS (<device>): xfs_log_force: error 5 returned.
> <5>XFS (<device>): xfs_do_force_shutdown(0x1) called from line 1063 of
> file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa0575
> <4>XFS (<device>): xfs_log_force: error 5 returned.
> <4>XFS (<device>): xfs_log_force: error 5 returned.
> ...
> <1>XFS (mtab~): metadata I/O error: block 0xa03e1b ("xlog_iodone")
> error 5 buf count 1024
And the rest of dmesg showing the stack trace of the oops?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: possible null pointer in xlog_iodone
2013-06-18 0:19 ` Dave Chinner
@ 2013-06-18 0:37 ` Vlad Bespalov
0 siblings, 0 replies; 6+ messages in thread
From: Vlad Bespalov @ 2013-06-18 0:37 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
---------------- system log:
<1>XFS (mtab~): metadata I/O error: block 0xa03e1b ("xlog_iodone")
error 5 buf count 1024
<1>BUG: unable to handle kernel paging request at 00000005000000fe
<1>IP: [<ffffffffa057b14e>] __xfs_printk+0xe/0x50 [xfs]
<4>PGD 0
<4>Oops: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/virtual/block/<device>/queue/hw_sector_size
<4>CPU 3
<4>Modules linked in: xfs exportfs fuse fiop(P)(U) sunrpc ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 uinput power_meter sg dcdbas microcode serio_raw
iTCO_wdt iTCO_vendor_support i7core_edac edac_core bnx2 ext4 mbcache
jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix
ocz10xx(U) mptsas mptscsih mptbase mpt2sas scsi_transport_sas
raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
speedstep_lib]
<4>
<4>Pid: 4408, comm: xfslogd/3 Tainted: P ---------------
2.6.32-358.2.1.el6.x86_64 #1 Dell Inc. PowerEdge T410/0H19HD
<4>RIP: 0010:[<ffffffffa057b14e>] [<ffffffffa057b14e>]
__xfs_printk+0xe/0x50 [xfs]
<4>RSP: 0018:ffff8807f99a1d30 EFLAGS: 00010206
<4>RAX: ffff8807f99a1d40 RBX: 0000000000000002 RCX: 0000000000000002
<4>RDX: ffff8807f99a1d60 RSI: 0000000500000006 RDI: ffffffffa058d4de
<4>RBP: ffff8807f99a1d30 R08: 000000000000041c R09: ffffffffa058c9e0
<4>R10: 0000000000000003 R11: 0000000000000000 R12: 0000000500000006
<4>R13: 0000000000000002 R14: 0000000000000000 R15: ffffe8fbe9a2d208
<4>FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 00000005000000fe CR3: 0000000001a85000 CR4: 00000000000007e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process xfslogd/3 (pid: 4408, threadinfo ffff8807f99a0000, task
ffff8807f8afeaa0)
<4>Stack:
<4> ffff8807f99a1da0 ffffffffa057b243 0000000000000010 ffff8807f99a1db0
<4><d> ffff8807f99a1d70 ffffffffa057b171 ffffffffa0589438 ffff8807f99a1d40
<4><d> ffff880700000030 ffff8807f99a1de0 ffffffffa0586570 0000000000000002
<4>Call Trace:
<4> [<ffffffffa057b243>] xfs_notice+0x53/0x60 [xfs]
<4> [<ffffffffa057b171>] ? __xfs_printk+0x31/0x50 [xfs]
<4> [<ffffffffa056fc6d>] xfs_do_force_shutdown+0xed/0x170 [xfs]
<4> [<ffffffffa0559e51>] ? xlog_iodone+0xa1/0xb0 [xfs]
<4> [<ffffffffa0559e51>] xlog_iodone+0xa1/0xb0 [xfs]
<4> [<ffffffffa0574c30>] ? xfs_buf_iodone_work+0x0/0x50 [xfs]
<4> [<ffffffffa0574c56>] xfs_buf_iodone_work+0x26/0x50 [xfs]
<4> [<ffffffff81090ae0>] worker_thread+0x170/0x2a0
<4> [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81090970>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff81096936>] kthread+0x96/0xa0
<4> [<ffffffff8100c0ca>] child_rip+0xa/0x20
<4> [<ffffffff810968a0>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
<4>Code: 58 a0 c7 44 24 08 01 00 00 00 e8 de 96 d0 e0 c9 c3 66 66 66
2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 48 85 f6 74 2a
<48> 8b 86 f8 00 00 00 48 85 c0 74 1e 48 89 d1 48 89 fe 48 89 c2
<1>RIP [<ffffffffa057b14e>] __xfs_printk+0xe/0x50 [xfs]
<4> RSP <ffff8807f99a1d30>
<4>CR2: 00000005000000fe
-------------------------------------------------------------
---------------- another system log:
<1>XFS (<device>): metadata I/O error: block 0xa01edc ("xlog_iodone")
error 5 buf count 1024
<5>XFS (<device>): xfs_do_force_shutdown(0x2) called from line 1052 of
file fs/xfs/xfs_log.c. Return address = 0xffffffffa0756e51
<1>XFS (<device>): Log I/O Error Detected. Shutting down filesystem
<1>XFS (<device>): Please umount the filesystem and rectify the problem(s)
<4>XFS (<device>): xfs_log_force: error 5 returned.
<5>XFS (<device>): xfs_do_force_shutdown(0x1) called from line 1063 of
file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa0772693
<4>XFS (<device>): xfs_log_force: error 5 returned.
<4>XFS (<device>): xfs_log_force: error 5 returned.
<5>sd 3:0:0:2: [sde] Synchronizing SCSI cache
...
<1>XFS: metadata I/O error: block 0xa01eda ("xlog_iodone") error 5 buf
count 1024
<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffffa0756e37>] xlog_iodone+0x87/0xb0 [xfs]
<4>PGD 3544bd067 PUD 246d05067 PMD 0
<4>Oops: 0000 [#1] SMP
<4>last sysfs file:
/sys/devices/pci0000:00/0000:00:0a.0/0000:02:00.0/host3/port-3:0/end_device-3:0/target3:0:0/3:0:0:2/delete
<4>CPU 1
<4>Modules linked in: xfs exportfs fiop(P)(U) sunrpc ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT
nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter
ip6_tables ipv6 uinput power_meter sg dcdbas microcode serio_raw
iTCO_wdt iTCO_vendor_support i7core_edac edac_core bnx2 ext4 mbcache
jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix
mptsas mptscsih mptbase ocz10xx(U) mpt2sas scsi_transport_sas
raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded:
fiop]
<4>
<4>Pid: 5552, comm: xfslogd/1 Tainted: P ---------------
2.6.32-358.2.1.el6.x86_64 #1 Dell Inc. PowerEdge T410/0H19HD
<4>RIP: 0010:[<ffffffffa0756e37>] [<ffffffffa0756e37>]
xlog_iodone+0x87/0xb0 [xfs]
<4>RSP: 0018:ffff88080c113df0 EFLAGS: 00010246
<4>RAX: 0000000000000055 RBX: ffff880828809980 RCX: 0000000000002f3d
<4>RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff880827243200
<4>RBP: ffff88080c113e10 R08: 0000000000042702 R09: 00000000fffffffb
<4>R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
<4>R13: ffff880827243200 R14: ffff88080c113fd8 R15: ffffe8fbe9a0cec8
<4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 000000027a961000 CR4: 00000000000007e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process xfslogd/1 (pid: 5552, threadinfo ffff88080c112000, task
ffff880826be2aa0)
<4>Stack:
<4> ffff88080c113e30 ffff880827243200 ffffe8fbe9a0cec0 ffffffffa0771c30
<4><d> ffff88080c113e30 ffffffffa0771c56 ffffe8fbe9a0cec0 ffffe8fbe9a0cec0
<4><d> ffff88080c113ee0 ffffffff81090ae0 0000000000000000 0000000000000000
<4>Call Trace:
<4> [<ffffffffa0771c30>] ? xfs_buf_iodone_work+0x0/0x50 [xfs]
<4> [<ffffffffa0771c56>] xfs_buf_iodone_work+0x26/0x50 [xfs]
<4> [<ffffffff81090ae0>] worker_thread+0x170/0x2a0
<4> [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81090970>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff81096936>] kthread+0x96/0xa0
<4> [<ffffffff8100c0ca>] child_rip+0xa/0x20
<4> [<ffffffff810968a0>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
<4>Code: e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 66 0f 1f 84 00 00 00 00 00
48 c7 c6 f8 2d 78 a0 4c 89 ef e8 c1 93 01 00 4c 89 ef e8 29 8a 01 00
<49> 8b 3c 24 be 02 00 00 00 b9 1c 04 00 00 48 c7 c2 e0 99 78 a0
<1>RIP [<ffffffffa0756e37>] xlog_iodone+0x87/0xb0 [xfs]
<4> RSP <ffff88080c113df0>
<4>CR2: 0000000000000000
---------------- crash core for 2nd case is available:
PID: 5552 TASK: ffff880826be2aa0 CPU: 1 COMMAND: "xfslogd/1"
#0 [ffff88080c1139e0] machine_kexec at ffffffff81035bbb
#1 [ffff88080c113a40] crash_kexec at ffffffff810c0dd2
#2 [ffff88080c113b10] oops_end at ffffffff81511450
#3 [ffff88080c113b40] no_context at ffffffff81046bfb
#4 [ffff88080c113b90] __bad_area_nosemaphore at ffffffff81046e85
#5 [ffff88080c113be0] bad_area_nosemaphore at ffffffff81046f53
#6 [ffff88080c113bf0] __do_page_fault at ffffffff810476b1
#7 [ffff88080c113d10] do_page_fault at ffffffff8151339e
#8 [ffff88080c113d40] page_fault at ffffffff81510755
[exception RIP: xlog_iodone+135]
RIP: ffffffffa0756e37 RSP: ffff88080c113df0 RFLAGS: 00010246
RAX: 0000000000000055 RBX: ffff880828809980 RCX: 0000000000002f3d
RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff880827243200
RBP: ffff88080c113e10 R8: 0000000000042702 R9: 00000000fffffffb
R10: 0000000000000003 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880827243200 R14: ffff88080c113fd8 R15: ffffe8fbe9a0cec8
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff88080c113e18] xfs_buf_iodone_work at ffffffffa0771c56 [xfs]
#10 [ffff88080c113e38] worker_thread at ffffffff81090ae0
#11 [ffff88080c113ee8] kthread at ffffffff81096936
#12 [ffff88080c113f48] kernel_thread at ffffffff8100c0ca
On 18 June 2013 04:19, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, Jun 17, 2013 at 06:38:56PM +0400, Vlad Bespalov wrote:
>> CentOS 6.4 kernel 2.6.32-358.2.1.el6.x86_64
>> i`ll try to make a script that can be used out-of-house.
>>
>> several requests may have failed (un)conveniently, so we have passed
>> through setting XLOG_IO_ERROR in xfs_log_force_unmount() and
>> eventually skipped xlog_wait altogether:
>> -----------------
>> <1>XFS (<device>): metadata I/O error: block 0xa03e1d ("xlog_iodone")
>> error 5 buf count 1024
>> <5>XFS (<device>): xfs_do_force_shutdown(0x2) called from line 1052 of
>> file fs/xfs/xfs_log.c. Return address = 0xffffffffa0559e51
>> <1>XFS (<device>): Log I/O Error Detected. Shutting down filesystem
>> <1>XFS (<device>): Please umount the filesystem and rectify the problem(s)
>> <4>XFS (<device>): xfs_log_force: error 5 returned.
>> <5>XFS (<device>): xfs_do_force_shutdown(0x1) called from line 1063 of
>> file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa0575
>> <4>XFS (<device>): xfs_log_force: error 5 returned.
>> <4>XFS (<device>): xfs_log_force: error 5 returned.
>> ...
>> <1>XFS (mtab~): metadata I/O error: block 0xa03e1b ("xlog_iodone")
>> error 5 buf count 1024
>
> And the rest of dmesg showing the stack trace of the oops?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-06-18 0:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-06-14 17:15 possible null pointer in xlog_iodone Vlad Bespalov
2013-06-14 19:42 ` Mark Tinguely
2013-06-15 1:03 ` Dave Chinner
2013-06-17 14:38 ` Vlad Bespalov
2013-06-18 0:19 ` Dave Chinner
2013-06-18 0:37 ` Vlad Bespalov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox