public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
* xfs umount with i/o error hang/memory corruption
@ 2014-04-04 18:15 Bob Mastors
  2014-04-04 18:50 ` Stan Hoeppner
  2014-04-04 21:20 ` Dave Chinner
  0 siblings, 2 replies; 6+ messages in thread
From: Bob Mastors @ 2014-04-04 18:15 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 2672 bytes --]

Greetings,

I am new to xfs and am running into a problem
and would appreciate any guidance on how to proceed.

After an i/o error from the block device that xfs is using,
an umount results in a message like:
[  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down filesystem
[  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
Please umount the filesystem and rectify the problem(s)
Note the garbage on the previous line which suggests memory corruption.
About half the time I get the garbled log message. About half the time
umount hangs.

And then I get this kind of error and the system is unresponsive:
Message from syslogd@debian at Apr  4 09:27:40 ...
 kernel:[  680.080022] BUG: soft lockup - CPU#2 stuck for 22s! [umount:2849]

The problem appears to be similar to this issue:
http://www.spinics.net/lists/linux-xfs/msg00061.html

I can reproduce the problem easily using open-iscsi to create
the block device with an iscsi initiator.
I use lio to create an iscsi target.

The problem is triggered by doing an iscsi logout which causes
the block device to return i/o errors to xfs.
Steps to reproduce the problem are below.

Using VirtualBox, I can reproduce it with two processors but not one.
I first saw this on a 3.8 kernel and most recently reproduced it with 3.14+.

Thanks,
Bob

virtual box setup
-----------------
1 GB ram
8 GB disk
2 processors
Operating System: Linux
Version: Debian (64 bit)
Bridged networking
iso: debian-7.4.0-amd64-netinst.iso
Software selection
    select SSH server
    select Standard system utilities
    deselect everything else

dhcp ip address: 172.17.5.129

upgrade linux to 3.14.0+
------------------------
ssh 172.17.5.129
su
cd /mnt/ksrc/linux
make modules_install
make install
reboot

install tools
-------------
ssh 172.17.5.129
su
apt-get install xfsprogs
apt-get install open-iscsi
apt-get install --no-install-recommends targetcli python-urwid
reboot

reproduce bug
-------------
ssh 172.17.5.129
su
targetcli
    cd backstores
    fileio/ create name=file_backend file_or_dev=my_disk size=100M
    /iscsi create
    portals/ create 172.17.5.129
    cd <
    luns/ create /backstores/fileio/file_backend
    cd <
    set attribute authentication=0 demo_mode_write_protect=0
generate_node_acls=1 cache_dynamic_acls=1
    exit

iscsiadm -m discovery -t sendtargets -p 172.17.5.129
iscsiadm -m node --login --target iqn.from_previous_iscsiadm_cmd_output
fdisk -l
mkfs -t xfs -f /dev/sdXX
mkdir /mnt/vola
mount -t xfs /dev/sdXX /mnt/vola
iscsiadm -m node -U all
umount /mnt/vola

[-- Attachment #1.2: Type: text/html, Size: 3723 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: xfs umount with i/o error hang/memory corruption
  2014-04-04 18:15 xfs umount with i/o error hang/memory corruption Bob Mastors
@ 2014-04-04 18:50 ` Stan Hoeppner
  2014-04-04 19:47   ` Bob Mastors
  2014-04-04 21:20 ` Dave Chinner
  1 sibling, 1 reply; 6+ messages in thread
From: Stan Hoeppner @ 2014-04-04 18:50 UTC (permalink / raw)
  To: Bob Mastors, xfs

On 4/4/2014 1:15 PM, Bob Mastors wrote:
> Greetings,
> 
> I am new to xfs and am running into a problem
> and would appreciate any guidance on how to proceed.
> 
> After an i/o error from the block device that xfs is using,
> an umount results in a message like:
> [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down filesystem
> [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> Please umount the filesystem and rectify the problem(s)
> Note the garbage on the previous line which suggests memory corruption.
> About half the time I get the garbled log message. About half the time
> umount hangs.
> 
> And then I get this kind of error and the system is unresponsive:
> Message from syslogd@debian at Apr  4 09:27:40 ...
>  kernel:[  680.080022] BUG: soft lockup - CPU#2 stuck for 22s! [umount:2849]
> 
> The problem appears to be similar to this issue:
> http://www.spinics.net/lists/linux-xfs/msg00061.html
> 
> I can reproduce the problem easily using open-iscsi to create
> the block device with an iscsi initiator.
> I use lio to create an iscsi target.
> 
> The problem is triggered by doing an iscsi logout which causes
> the block device to return i/o errors to xfs.
> Steps to reproduce the problem are below.

This is not a problem but the expected behavior.  XFS is designed to do
this to prevent filesystem corruption.  Logging out of a LUN is no
different than pulling the power plug on a direct attached disk drive.
Surely you would not do that to a running filesystem.

> Using VirtualBox, I can reproduce it with two processors but not one.
> I first saw this on a 3.8 kernel and most recently reproduced it with 3.14+.
...

The only problem I see here is that XFS should be shutting down every
time the disk device disappears.  Which means in your test cases where
it does not, your VM environment isn't passing the IO errors up the
stack, and it should be.  Which means your VM environment is broken.

Cheers,

Stan


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: xfs umount with i/o error hang/memory corruption
  2014-04-04 18:50 ` Stan Hoeppner
@ 2014-04-04 19:47   ` Bob Mastors
  0 siblings, 0 replies; 6+ messages in thread
From: Bob Mastors @ 2014-04-04 19:47 UTC (permalink / raw)
  To: stan; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 3066 bytes --]

On Fri, Apr 4, 2014 at 12:50 PM, Stan Hoeppner <stan@hardwarefreak.com>wrote:

> On 4/4/2014 1:15 PM, Bob Mastors wrote:
> > Greetings,
> >
> > I am new to xfs and am running into a problem
> > and would appreciate any guidance on how to proceed.
> >
> > After an i/o error from the block device that xfs is using,
> > an umount results in a message like:
> > [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down
> filesystem
> > [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> > Please umount the filesystem and rectify the problem(s)
> > Note the garbage on the previous line which suggests memory corruption.
> > About half the time I get the garbled log message. About half the time
> > umount hangs.
> >
> > And then I get this kind of error and the system is unresponsive:
> > Message from syslogd@debian at Apr  4 09:27:40 ...
> >  kernel:[  680.080022] BUG: soft lockup - CPU#2 stuck for 22s!
> [umount:2849]
> >
> > The problem appears to be similar to this issue:
> > http://www.spinics.net/lists/linux-xfs/msg00061.html
> >
> > I can reproduce the problem easily using open-iscsi to create
> > the block device with an iscsi initiator.
> > I use lio to create an iscsi target.
> >
> > The problem is triggered by doing an iscsi logout which causes
> > the block device to return i/o errors to xfs.
> > Steps to reproduce the problem are below.
>
> This is not a problem but the expected behavior.  XFS is designed to do
> this to prevent filesystem corruption.  Logging out of a LUN is no
> different than pulling the power plug on a direct attached disk drive.
> Surely you would not do that to a running filesystem.
>
Sorry, I don't think I was clear on the nature of the problem and it's wide
ranging effects.
Using iscsi to access block storage on another server is the goal.
A failure on the other server is possible which could result in the iscsi
initiator returning i/o errors to xfs.

The behavior that I would like from xfs is to put the filesystem in some
kind of offline state
on an i/o error from the block device. Today xfs usually has this desirable
behavior.
But there is a corner case were instead xfs hangs the entire server,
forcing a hard reboot.

On a large file server with many filesystems using iscsi to talk to block
storage on multiple
block servers, I would like the failure of a single block server to only
impact the filesystems
that are dependent on the block server, and to not impact the other
filesystems.

Bob

> Using VirtualBox, I can reproduce it with two processors but not one.
> > I first saw this on a 3.8 kernel and most recently reproduced it with
> 3.14+.
> ...
>
> The only problem I see here is that XFS should be shutting down every
> time the disk device disappears.  Which means in your test cases where
> it does not, your VM environment isn't passing the IO errors up the
> stack, and it should be.  Which means your VM environment is broken.
>
> Cheers,
>
> Stan
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3963 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: xfs umount with i/o error hang/memory corruption
  2014-04-04 18:15 xfs umount with i/o error hang/memory corruption Bob Mastors
  2014-04-04 18:50 ` Stan Hoeppner
@ 2014-04-04 21:20 ` Dave Chinner
  2014-04-04 21:40   ` Bob Mastors
  1 sibling, 1 reply; 6+ messages in thread
From: Dave Chinner @ 2014-04-04 21:20 UTC (permalink / raw)
  To: Bob Mastors; +Cc: xfs

On Fri, Apr 04, 2014 at 12:15:23PM -0600, Bob Mastors wrote:
> Greetings,
> 
> I am new to xfs and am running into a problem
> and would appreciate any guidance on how to proceed.
> 
> After an i/o error from the block device that xfs is using,
> an umount results in a message like:
> [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down filesystem
> [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> Please umount the filesystem and rectify the problem(s)
> Note the garbage on the previous line which suggests memory corruption.
> About half the time I get the garbled log message. About half the time
> umount hangs.

I got an email about this last night with a different trigger - thin
provisioning failing log IO in the unmount path. I know what the
problem is, I just don't have a fix for it yet.

To confirm it's the same problem, can you post the entirity of the
dmesg where the error occurs?

In essence, the log IO failure is triggering a shutdown, and as part
of the shutdown process it wakes anyone waiting on a log force.
The log quiesce code that waits for log completion during unmount
uses a log force to ensure the log is idle before tearing down all
the log structures and finishing the unmount. Unfortunatey, the log
force the unmount blocks on is woken prematurely by the shutdown,
and hence it runs before the log IO processing is completed. Hence
the use after free.

> And then I get this kind of error and the system is unresponsive:
> Message from syslogd@debian at Apr  4 09:27:40 ...
>  kernel:[  680.080022] BUG: soft lockup - CPU#2 stuck for 22s! [umount:2849]
> 
> The problem appears to be similar to this issue:
> http://www.spinics.net/lists/linux-xfs/msg00061.html

Similar symptoms, but not the same bug.

> The problem is triggered by doing an iscsi logout which causes
> the block device to return i/o errors to xfs.
> Steps to reproduce the problem are below.

Seems similar to the thinp ENOSPC issue I mentioned above - data IO
errors occur, then you do an unmount, which causes a log IO error
writing the superblock, and then this happens....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: xfs umount with i/o error hang/memory corruption
  2014-04-04 21:20 ` Dave Chinner
@ 2014-04-04 21:40   ` Bob Mastors
  2014-04-04 21:57     ` Dave Chinner
  0 siblings, 1 reply; 6+ messages in thread
From: Bob Mastors @ 2014-04-04 21:40 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 10826 bytes --]

On Fri, Apr 4, 2014 at 3:20 PM, Dave Chinner <david@fromorbit.com> wrote:

> On Fri, Apr 04, 2014 at 12:15:23PM -0600, Bob Mastors wrote:
> > Greetings,
> >
> > I am new to xfs and am running into a problem
> > and would appreciate any guidance on how to proceed.
> >
> > After an i/o error from the block device that xfs is using,
> > an umount results in a message like:
> > [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down
> filesystem
> > [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> > Please umount the filesystem and rectify the problem(s)
> > Note the garbage on the previous line which suggests memory corruption.
> > About half the time I get the garbled log message. About half the time
> > umount hangs.
>
> I got an email about this last night with a different trigger - thin
> provisioning failing log IO in the unmount path. I know what the
> problem is, I just don't have a fix for it yet.
>
> To confirm it's the same problem, can you post the entirity of the
> dmesg where the error occurs?
>
Here is the console output from putting a BUG() call  in
xfs_do_force_shutdown
after the "Please umount ..." message.
Let me know if want something else.

I don't know how much help I can be on this, but I can run more tests
or try out fixes.

Thanks,
Bob

[  111.004665] XFS (sdb): Mounting Filesystem
[  111.099442] XFS (sdb): Ending clean mount
[  129.548644] XFS (sdb): metadata I/O error: block 0x19022 ("xlog_iodone")
error 19 numblks 64
[  129.618739] XFS (sdb): xfs_do_force_shutdown(0x2) called from line 1170
of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa04c8001
[  129.699768] XFS (sdb): Log I/O Error Detected.  Shutting down filesystem
[  129.699924] XFS (sdb): Unable to update superblock counters. Freespace
may not be correct on next mount.
[  129.699927] XFS (sdb): xfs_log_force: error 5 returned.
[  129.716031] XFS (P専/): Please umount the filesystem and rectify the
problem(s)
[  129.718411] ------------[ cut here ]------------
[  129.719120] Kernel BUG at ffffffffa04726ac [verbose debug info
unavailable]
[  129.720016] invalid opcode: 0000 [#1] SMP
[  129.720016] Modules linked in: xfs(F) libcrc32c ib_srpt tcm_qla2xxx
qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod
target_core_pscsi target_core_file target_core_iblock target_core_mod
configfs ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs_acl nfs lockd fscac\
he sunrpc snd_intel8x0 snd_ac97_codec parport_pc parport ac97_bus snd_pcm
psmouse snd_timer serio_raw snd joydev evbug mac_hid pcspkr i2c_piix4
soundcore hid_generic usbhid hid e1000 ahci libahci
[  129.720016] CPU: 1 PID: 234 Comm: kworker/1:1H Tainted: GF
 3.14.0+ #1
[  129.720016] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox
12/01/2006
[  129.720016] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
[  129.720016] task: ffff88002f598000 ti: ffff88002fecc000 task.ti:
ffff88002fecc000
[  129.720016] RIP: 0010:[<ffffffffa04726ac>]  [<ffffffffa04726ac>]
xfs_do_force_shutdown+0x9c/0x170 [xfs]
[  129.720016] RSP: 0018:ffff88002fecdd68  EFLAGS: 00010296
[  129.720016] RAX: 0000000000000043 RBX: 0000000000000002 RCX:
0000000000000000
[  129.720016] RDX: ffff88003fd0efb0 RSI: ffff88003fd0d3d8 RDI:
ffff88003fd0d3d8
[  129.720016] RBP: ffff88002fecdd98 R08: 0000000000000000 R09:
0000000000000000
[  129.720016] R10: 00000000000001cf R11: 00000000000001ce R12:
ffff880028865000
[  129.720016] R13: 0000000000000002 R14: 0000000000000000 R15:
0000000000000000
[  129.720016] FS:  0000000000000000(0000) GS:ffff88003fd00000(0000)
knlGS:0000000000000000
[  129.720016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  129.720016] CR2: 00000000006105e8 CR3: 0000000001c0e000 CR4:
00000000000006e0
[  129.720016] Stack:
[  129.720016]  ffffffffa04c8001 0000000000000000 ffff880030140300
ffff88003026dc00
[  129.720016]  ffff88002fa13980 0000000000000000 ffff88002fecddc8
ffffffffa04c8001
[  129.720016]  ffff88003653cf48 ffff88002fa13a20 ffff88002fa13980
ffffe8ffffd01e00
[  129.720016] Call Trace:
[  129.720016]  [<ffffffffa04c8001>] ? xlog_iodone+0x91/0xa0 [xfs]
[  129.720016]  [<ffffffffa04c8001>] xlog_iodone+0x91/0xa0 [xfs]
[  129.720016]  [<ffffffffa046ac3e>] xfs_buf_iodone_work+0x5e/0xc0 [xfs]
[  129.720016]  [<ffffffff8106ad04>] process_one_work+0x174/0x4a0
[  129.720016]  [<ffffffff8106c0ab>] worker_thread+0x11b/0x370
[  129.720016]  [<ffffffff8106bf90>] ? manage_workers.isra.20+0x2d0/0x2d0
[  129.720016]  [<ffffffff81072c29>] kthread+0xc9/0xe0
[  129.720016]  [<ffffffff81072b60>] ? flush_kthread_worker+0xb0/0xb0
[  129.720016]  [<ffffffff816d6aec>] ret_from_fork+0x7c/0xb0
[  129.720016]  [<ffffffff81072b60>] ? flush_kthread_worker+0xb0/0xb0
[  129.720016] Code: d0 45 85 ed 0f 85 a5 00 00 00 f6 c3 20 66 90 75 7e 83
e3 10 0f 84 b5 00 00 00 48 c7 c6 18 8c 4e a0 4c 89 e7 31 c0 e8 54 92 00 00
<0f> 0b 66 90 48 8b 45 08 49 89 d1 41 89 c8 48 c7 c2 d0 4f 4e a0
[  129.720016] RIP  [<ffffffffa04726ac>] xfs_do_force_shutdown+0x9c/0x170
[xfs]
[  129.720016]  RSP <ffff88002fecdd68>
[  129.777164] ---[ end trace d335180e3c345156 ]---
[  129.777953] BUG: unable to handle kernel paging request at
ffffffffffffffd8
[  129.779087] IP: [<ffffffff81072c80>] kthread_data+0x10/0x20
[  129.779960] PGD 1c0f067 PUD 1c11067 PMD 0
[  129.780900] Oops: 0000 [#2] SMP
[  129.781150] Modules linked in: xfs(F) libcrc32c ib_srpt tcm_qla2xxx
qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod
target_core_pscsi target_core_file target_core_iblock target_core_mod
configfs ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs_acl nfs lockd fscac\
he sunrpc snd_intel8x0 snd_ac97_codec parport_pc parport ac97_bus snd_pcm
psmouse snd_timer serio_raw snd joydev evbug mac_hid pcspkr i2c_piix4
soundcore hid_generic usbhid hid e1000 ahci libahci
[  129.781150] CPU: 1 PID: 234 Comm: kworker/1:1H Tainted: GF     D
 3.14.0+ #1
[  129.781150] Hardware name: innotek GmbH VirtualBox, BIOS VirtualBox
12/01/2006
[  129.781150] task: ffff88002f598000 ti: ffff88002fecc000 task.ti:
ffff88002fecc000
[  129.781150] RIP: 0010:[<ffffffff81072c80>]  [<ffffffff81072c80>]
kthread_data+0x10/0x20
[  129.781150] RSP: 0018:ffff88002fecd9f8  EFLAGS: 00010046
[  129.781150] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
ffffffff81e75380
[  129.781150] RDX: 0000000000000001 RSI: 0000000000000001 RDI:
ffff88002f598000
[  129.781150] RBP: ffff88002fecd9f8 R08: 0000000000000000 R09:
0000000000000164
[  129.781150] R10: ffffffff8104c8b4 R11: 0000000000000000 R12:
0000000000000001
[  129.781150] R13: ffff88002f5984e0 R14: ffff88003d9f8000 R15:
ffff88002f5983d8
[  129.781150] FS:  0000000000000000(0000) GS:ffff88003fd00000(0000)
knlGS:0000000000000000
[  129.781150] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  129.781150] CR2: 0000000000000028 CR3: 0000000001c0e000 CR4:
00000000000006e0
[  129.781150] Stack:
[  129.781150]  ffff88002fecda18 ffffffff8106c526 ffff88003fd13140
0000000000000001
[  129.781150]  ffff88002fecda98 ffffffff816ca7cf 0000000000000001
ffff88002f598000
[  129.781150]  ffff88002f598000 0000000000013140 ffff88002fecdfd8
0000000000013140
[  129.781150] Call Trace:
[  129.781150]  [<ffffffff8106c526>] wq_worker_sleeping+0x16/0xa0
[  129.781150]  [<ffffffff816ca7cf>] __schedule+0x56f/0x7c0
[  129.781150]  [<ffffffff816caaf9>] schedule+0x29/0x70
[  129.781150]  [<ffffffff81052075>] do_exit+0x6e5/0xa70
[  129.781150]  [<ffffffff816c2303>] ? printk+0x4d/0x4f
[  129.781150]  [<ffffffff816cf0e1>] oops_end+0xa1/0xf0
[  129.781150]  [<ffffffff81006208>] die+0x58/0x90
[  129.781150]  [<ffffffff816ce93b>] do_trap+0xcb/0x170
[  129.781150]  [<ffffffff8100308d>] do_invalid_op+0xad/0x100
[  129.781150]  [<ffffffffa04726ac>] ? xfs_do_force_shutdown+0x9c/0x170
[xfs]
[  129.781150]  [<ffffffff816c2303>] ? printk+0x4d/0x4f
[  129.781150]  [<ffffffff816d8088>] invalid_op+0x18/0x20
[  129.781150]  [<ffffffffa04726ac>] ? xfs_do_force_shutdown+0x9c/0x170
[xfs]
[  129.781150]  [<ffffffffa04726ac>] ? xfs_do_force_shutdown+0x9c/0x170
[xfs]
[  129.781150]  [<ffffffffa04c8001>] ? xlog_iodone+0x91/0xa0 [xfs]
[  129.781150]  [<ffffffffa04c8001>] xlog_iodone+0x91/0xa0 [xfs]
[  129.781150]  [<ffffffffa046ac3e>] xfs_buf_iodone_work+0x5e/0xc0 [xfs]
[  129.781150]  [<ffffffff8106ad04>] process_one_work+0x174/0x4a0
[  129.781150]  [<ffffffff8106c0ab>] worker_thread+0x11b/0x370
[  129.781150]  [<ffffffff8106bf90>] ? manage_workers.isra.20+0x2d0/0x2d0
[  129.781150]  [<ffffffff81072c29>] kthread+0xc9/0xe0
[  129.781150]  [<ffffffff81072b60>] ? flush_kthread_worker+0xb0/0xb0
[  129.781150]  [<ffffffff816d6aec>] ret_from_fork+0x7c/0xb0
[  129.781150]  [<ffffffff81072b60>] ? flush_kthread_worker+0xb0/0xb0
[  129.781150] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66
2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 88 04 00 00 55 48 89 e5
<48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[  129.781150] RIP  [<ffffffff81072c80>] kthread_data+0x10/0x20
[  129.781150]  RSP <ffff88002fecd9f8>
[  129.781150] CR2: ffffffffffffffd8
[  129.781150] ---[ end trace d335180e3c345157 ]---
[  129.781150] Fixing recursive fault but reboot is needed!


> In essence, the log IO failure is triggering a shutdown, and as part
> of the shutdown process it wakes anyone waiting on a log force.
> The log quiesce code that waits for log completion during unmount
> uses a log force to ensure the log is idle before tearing down all
> the log structures and finishing the unmount. Unfortunatey, the log
> force the unmount blocks on is woken prematurely by the shutdown,
> and hence it runs before the log IO processing is completed. Hence
> the use after free.
>
> > And then I get this kind of error and the system is unresponsive:
> > Message from syslogd@debian at Apr  4 09:27:40 ...
> >  kernel:[  680.080022] BUG: soft lockup - CPU#2 stuck for 22s!
> [umount:2849]
> >
> > The problem appears to be similar to this issue:
> > http://www.spinics.net/lists/linux-xfs/msg00061.html
>
> Similar symptoms, but not the same bug.
>
> > The problem is triggered by doing an iscsi logout which causes
> > the block device to return i/o errors to xfs.
> > Steps to reproduce the problem are below.
>
> Seems similar to the thinp ENOSPC issue I mentioned above - data IO
> errors occur, then you do an unmount, which causes a log IO error
> writing the superblock, and then this happens....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

[-- Attachment #1.2: Type: text/html, Size: 13324 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: xfs umount with i/o error hang/memory corruption
  2014-04-04 21:40   ` Bob Mastors
@ 2014-04-04 21:57     ` Dave Chinner
  0 siblings, 0 replies; 6+ messages in thread
From: Dave Chinner @ 2014-04-04 21:57 UTC (permalink / raw)
  To: Bob Mastors; +Cc: xfs

On Fri, Apr 04, 2014 at 03:40:38PM -0600, Bob Mastors wrote:
> On Fri, Apr 4, 2014 at 3:20 PM, Dave Chinner <david@fromorbit.com> wrote:
> 
> > On Fri, Apr 04, 2014 at 12:15:23PM -0600, Bob Mastors wrote:
> > > Greetings,
> > >
> > > I am new to xfs and am running into a problem
> > > and would appreciate any guidance on how to proceed.
> > >
> > > After an i/o error from the block device that xfs is using,
> > > an umount results in a message like:
> > > [  370.636473] XFS (sdx): Log I/O Error Detected.  Shutting down
> > filesystem
> > > [  370.644073] XFS (h           ���h"h          ���H#h          ���bsg):
> > > Please umount the filesystem and rectify the problem(s)
> > > Note the garbage on the previous line which suggests memory corruption.
> > > About half the time I get the garbled log message. About half the time
> > > umount hangs.
> >
> > I got an email about this last night with a different trigger - thin
> > provisioning failing log IO in the unmount path. I know what the
> > problem is, I just don't have a fix for it yet.
> >
> > To confirm it's the same problem, can you post the entirity of the
> > dmesg where the error occurs?
> >
> Here is the console output from putting a BUG() call  in
> xfs_do_force_shutdown
> after the "Please umount ..." message.
> Let me know if want something else.
> 
> I don't know how much help I can be on this, but I can run more tests
> or try out fixes.

I'll let you know when I have a fix for you to try - probably early
next week as it's Saturday morning right now...

> [  129.548644] XFS (sdb): metadata I/O error: block 0x19022 ("xlog_iodone")
> error 19 numblks 64
> [  129.618739] XFS (sdb): xfs_do_force_shutdown(0x2) called from line 1170
> of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa04c8001
> [  129.699768] XFS (sdb): Log I/O Error Detected.  Shutting down filesystem
> [  129.699924] XFS (sdb): Unable to update superblock counters. Freespace
> may not be correct on next mount.
> [  129.699927] XFS (sdb): xfs_log_force: error 5 returned.
> [  129.716031] XFS (P専/): Please umount the filesystem and rectify the

Yup, that is identical to the bug report I got from the dm-thinp
folks. Thanks for the quick followup!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-04-04 21:57 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-04-04 18:15 xfs umount with i/o error hang/memory corruption Bob Mastors
2014-04-04 18:50 ` Stan Hoeppner
2014-04-04 19:47   ` Bob Mastors
2014-04-04 21:20 ` Dave Chinner
2014-04-04 21:40   ` Bob Mastors
2014-04-04 21:57     ` Dave Chinner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox