* "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors
@ 2015-06-24 7:39 Rasmus Borup Hansen
2015-06-25 16:41 ` Emmanuel Florac
2015-06-29 21:50 ` Dave Chinner
0 siblings, 2 replies; 15+ messages in thread
From: Rasmus Borup Hansen @ 2015-06-24 7:39 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 8109 bytes --]
Hi! Yesterday I got the following error messages from the kernel (Ubuntu trusty, 3.13.0-55):
[601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
[601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
[601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[601201.820013] XFS (dm-0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00996f0
[601201.820910] CPU: 1 PID: 421 Comm: xfsaild/dm-0 Not tainted 3.13.0-55-generic #92-Ubuntu
[601201.820913] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011
[601201.820914] 0000000000000001 ffff880035cc3bd0 ffffffff81723294 ffff8802ac71d800
[601201.820918] ffff880035cc3be8 ffffffffa009d6fb ffffffffa00996f0 ffff880035cc3c20
[601201.820919] ffffffffa009d755 000000d800205500 ffff88001aa2a400 ffff880050acc2b8
[601201.820922] Call Trace:
[601201.820930] [<ffffffff81723294>] dump_stack+0x45/0x56
[601201.820964] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]
[601201.820974] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.820984] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]
[601201.820997] [<ffffffffa00bab50>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[601201.821007] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821016] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821026] [<ffffffffa00996f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821030] [<ffffffff8109abc0>] ? wake_up_state+0x20/0x20
[601201.821040] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821050] [<ffffffffa009b336>] xfs_buf_iorequest+0x46/0x90 [xfs]
[601201.821060] [<ffffffffa009b3d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821070] [<ffffffffa009b56b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs]
[601201.821081] [<ffffffffa009c000>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821100] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821110] [<ffffffffa009c000>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821127] [<ffffffffa00facd7>] xfsaild+0x237/0x5c0 [xfs]
[601201.821145] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821148] [<ffffffff8108b702>] kthread+0xd2/0xf0
[601201.821150] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821153] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90
[601201.821155] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821158] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[601201.821581] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa009971c
[601201.832780] XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem
[601201.833292] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[601201.843375] ffff880238b00000: 58 44 32 42 01 00 0e c0 00 00 00 00 00 00 00 00 XD2B............
[601201.843924] ffff880238b00010: 00 00 00 20 3d a7 0c 7b 01 2e 8d 9e 01 88 00 10 ... =..{........
[601201.844472] ffff880238b00020: 00 00 00 19 21 a9 2d 2f 02 2e 2e 9e 01 88 00 20 ....!.-/.......
[601201.845031] ffff880238b00030: 00 00 00 07 39 2b 68 20 18 4d 4f 42 31 41 5f 48 ....9+h .MOB1A_H
[601201.845585] XFS (dm-0): Internal error xfs_da3_node_read_verify at line 240 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_da_btree.c. Caller 0xffffffffa009a885
[601201.858885] CPU: 1 PID: 2806 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #92-Ubuntu
[601201.858889] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011
[601201.858938] Workqueue: xfslogd xfs_buf_iodone_work [xfs]
[601201.858941] 0000000000000001 ffff8802af9cbd68 ffffffff81723294 ffff8802ac71d800
[601201.858944] ffff8802af9cbd80 ffffffffa009d6fb ffffffffa009a885 ffff8802af9cbdb8
[601201.858945] ffffffffa009d755 000000f0810a0465 ffff880031b35b00 ffff880238b00000
[601201.858948] Call Trace:
[601201.858958] [<ffffffff81723294>] dump_stack+0x45/0x56
[601201.858969] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]
[601201.858979] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.858989] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]
[601201.859005] [<ffffffffa00d146c>] xfs_da3_node_read_verify+0x8c/0x180 [xfs]
[601201.859014] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.859024] [<ffffffffa009a885>] xfs_buf_iodone_work+0x85/0xf0 [xfs]
[601201.859028] [<ffffffff81083b22>] process_one_work+0x182/0x450
[601201.859030] [<ffffffff81084911>] worker_thread+0x121/0x410
[601201.859032] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430
[601201.859035] [<ffffffff8108b702>] kthread+0xd2/0xf0
[601201.859037] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.859040] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90
[601201.859042] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.859044] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[601201.872288] XFS (dm-0): metadata I/O error: block 0x101ed393c0 ("xfs_trans_read_buf_map") error 117 numblks 8
[601201.942054] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
[601209.420268] XFS (dm-0): xfs_log_force: error 5 returned.
[601239.529710] XFS (dm-0): xfs_log_force: error 5 returned.
[601269.639150] XFS (dm-0): xfs_log_force: error 5 returned.
[601299.748593] XFS (dm-0): xfs_log_force: error 5 returned.
[601329.858034] XFS (dm-0): xfs_log_force: error 5 returned.
I'm currently running xfs_repair -n and so far I've seen the following output:
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
doubling cache size to 591200
directory flags set on non-directory inode 206624
directory flags set on non-directory inode 206625
directory flags set on non-directory inode 206626
directory flags set on non-directory inode 206627
directory flags set on non-directory inode 206628
directory flags set on non-directory inode 206629
directory flags set on non-directory inode 206630
directory flags set on non-directory inode 206631
directory flags set on non-directory inode 206632
directory flags set on non-directory inode 206633
directory flags set on non-directory inode 206634
Only the first 20 lines are included. There are currently 250000+ more lines with "directory flags set on non-directory inode" and the check is still running (the mostly small files take up around 30 TB, so it'll probably take a while).
I recently enabled user and project quota and updated from 3.13.0-53. The file system has been heavily used for the last month or so.
Does anyone have any thoughts on this? I'm tempted to stop using quotas when the file system (hopefully) works again, as it's my impression that project quotas are not widely used.
Best,
Rasmus
Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.
-----------------------------------------------------------------
Hansen, Rasmus Borup Intomics - from data to biology
System Administrator Diplomvej 377
Scientific Programmer DK-2800 Kgs. Lyngby
Denmark
E: rbh@intomics.com W: http://www.intomics.com/
P: +45 5167 7972 P: +45 8880 7979
[-- Attachment #1.2: Type: text/html, Size: 15759 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-06-24 7:39 "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors Rasmus Borup Hansen @ 2015-06-25 16:41 ` Emmanuel Florac 2015-06-26 6:14 ` Rasmus Borup Hansen 2015-06-29 21:50 ` Dave Chinner 1 sibling, 1 reply; 15+ messages in thread From: Emmanuel Florac @ 2015-06-25 16:41 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs Le Wed, 24 Jun 2015 09:39:45 +0200 Rasmus Borup Hansen <rbh@intomics.com> écrivait: > Only the first 20 lines are included. There are currently 250000+ > more lines with "directory flags set on non-directory inode" and the > check is still running (the mostly small files take up around 30 TB, > so it'll probably take a while). > > I recently enabled user and project quota and updated from 3.13.0-53. > The file system has been heavily used for the last month or so. > > Does anyone have any thoughts on this? I'm tempted to stop using > quotas when the file system (hopefully) works again, as it's my > impression that project quotas are not widely used. Did you first try remounting then unmounting the volume to clear the log? That could clear out xfs_repair output. -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-06-25 16:41 ` Emmanuel Florac @ 2015-06-26 6:14 ` Rasmus Borup Hansen 2015-07-02 7:58 ` Rasmus Borup Hansen 0 siblings, 1 reply; 15+ messages in thread From: Rasmus Borup Hansen @ 2015-06-26 6:14 UTC (permalink / raw) To: Emmanuel Florac; +Cc: xfs [-- Attachment #1.1: Type: text/plain, Size: 2619 bytes --] I tried mounting the file system read-only (which triggered a quota check – does this make sense when the file system is read-only?) and then I scanned the file system for files with the spurious inodes to see if I could find a pattern. This took quite a while, and I didn't find any patterns, except that all the files in directories with project quotas were affected (but there were also other files). I'm now running xfs_repair without -n (and I had to mount and unmount the file system before it would start). I'll report back when it has finished. Best, Rasmus Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. ----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark E: rbh@intomics.com W: http://www.intomics.com/ P: +45 5167 7972 P: +45 8880 7979 > On 25 Jun 2015, at 18:41, Emmanuel Florac <eflorac@intellique.com> wrote: > > Le Wed, 24 Jun 2015 09:39:45 +0200 > Rasmus Borup Hansen <rbh@intomics.com> écrivait: > >> Only the first 20 lines are included. There are currently 250000+ >> more lines with "directory flags set on non-directory inode" and the >> check is still running (the mostly small files take up around 30 TB, >> so it'll probably take a while). >> >> I recently enabled user and project quota and updated from 3.13.0-53. >> The file system has been heavily used for the last month or so. >> >> Does anyone have any thoughts on this? I'm tempted to stop using >> quotas when the file system (hopefully) works again, as it's my >> impression that project quotas are not widely used. > > Did you first try remounting then unmounting the volume to clear the > log? That could clear out xfs_repair output. > > -- > ------------------------------------------------------------------------ > Emmanuel Florac | Direction technique > | Intellique > | <eflorac@intellique.com> > | +33 1 78 94 84 02 > ------------------------------------------------------------------------ > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs [-- Attachment #1.2: Type: text/html, Size: 6196 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-06-26 6:14 ` Rasmus Borup Hansen @ 2015-07-02 7:58 ` Rasmus Borup Hansen 2015-07-02 9:26 ` Emmanuel Florac 0 siblings, 1 reply; 15+ messages in thread From: Rasmus Borup Hansen @ 2015-07-02 7:58 UTC (permalink / raw) To: xfs [-- Attachment #1.1: Type: text/plain, Size: 8083 bytes --] When I tried mounting the file system after running xfs_repair, a quota check was started. However it did not finish after a few hours like the quota check that was started when mounting the file system read only, so after a few days of waiting (with next to no disk activity) I restarted the server and mounted the file system without quotas. Soon afterwards I got the following error (note that it's a different line number in xfs_attr_leaf.c than the one I initially saw): [ 327.670974] ffff8802a273d000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ [ 327.678213] ffff8802a273d010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... [ 327.685878] ffff8802a273d020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 327.693950] ffff8802a273d030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 327.701567] XFS (dm-0): Internal error xfs_attr3_leaf_read_verify at line 246 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00cb885 [ 327.718829] CPU: 1 PID: 2805 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #94-Ubuntu [ 327.718830] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011 [ 327.718867] Workqueue: xfslogd xfs_buf_iodone_work [xfs] [ 327.718869] 0000000000000001 ffff8800b82bfd68 ffffffff81723294 ffff8800368dd800 [ 327.718875] ffff8800b82bfd80 ffffffffa00ce6fb ffffffffa00cb885 ffff8800b82bfdb8 [ 327.718877] ffffffffa00ce755 000000f600203100 ffff8802a140ad00 ffff8800368dd800 [ 327.718879] Call Trace: [ 327.718887] [<ffffffff81723294>] dump_stack+0x45/0x56 [ 327.718901] [<ffffffffa00ce6fb>] xfs_error_report+0x3b/0x40 [xfs] [ 327.718911] [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [ 327.718921] [<ffffffffa00ce755>] xfs_corruption_error+0x55/0x80 [xfs] [ 327.718935] [<ffffffffa00ebbdd>] xfs_attr3_leaf_read_verify+0x6d/0xf0 [xfs] [ 327.718945] [<ffffffffa00cb885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [ 327.718954] [<ffffffffa00cb885>] xfs_buf_iodone_work+0x85/0xf0 [xfs] [ 327.718958] [<ffffffff81083b22>] process_one_work+0x182/0x450 [ 327.718961] [<ffffffff81084911>] worker_thread+0x121/0x410 [ 327.718963] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430 [ 327.718965] [<ffffffff8108b702>] kthread+0xd2/0xf0 [ 327.718967] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [ 327.718970] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90 [ 327.718972] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [ 327.718973] XFS (dm-0): Corruption detected. Unmount and run xfs_repair [ 327.729003] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8 I started another xfs_repair which did not report any errors. After mounting the file system again (still without quotas) I discovered that ls would write "Structure needs cleaning" whenever it listed a certain file (and the kernel would output error messages like those above). This was a file I didn't need, so I tried deleting it and running yet another xfs_repair: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... agi unlinked bucket 11 is 2949684875 in ag 2 (inode=11539619467) - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 doubling cache size to 591232 - agno = 1 - agno = 2 - agno = 3 ... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... disconnected inode 11539619467, moving to lost+found Phase 7 - verify and correct link counts... done The file then turns up in lost+found and when I "ls" it I get the same errors again. I've tried deleting it from lost+found, but then xfs_repair finds it again with exactly the same output as show above and puts it back. Apart from that, everything apparently works fine. Is there a way to permanently get rid of the file in lost+found? Its size is apparently 0 bytes. Best, Rasmus Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. ----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark E: rbh@intomics.com W: http://www.intomics.com/ P: +45 5167 7972 P: +45 8880 7979 > On 26 Jun 2015, at 08:14, Rasmus Borup Hansen <rbh@intomics.com> wrote: > > I tried mounting the file system read-only (which triggered a quota check – does this make sense when the file system is read-only?) and then I scanned the file system for files with the spurious inodes to see if I could find a pattern. This took quite a while, and I didn't find any patterns, except that all the files in directories with project quotas were affected (but there were also other files). I'm now running xfs_repair without -n (and I had to mount and unmount the file system before it would start). I'll report back when it has finished. > > Best, > > Rasmus > > Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. > ----------------------------------------------------------------- > Hansen, Rasmus Borup Intomics - from data to biology > System Administrator Diplomvej 377 > Scientific Programmer DK-2800 Kgs. Lyngby > Denmark > E: rbh@intomics.com <mailto:rbh@intomics.com> W: http://www.intomics.com/ <http://www.intomics.com/> > P: +45 5167 7972 P: +45 8880 7979 > >> On 25 Jun 2015, at 18:41, Emmanuel Florac <eflorac@intellique.com <mailto:eflorac@intellique.com>> wrote: >> >> Le Wed, 24 Jun 2015 09:39:45 +0200 >> Rasmus Borup Hansen <rbh@intomics.com <mailto:rbh@intomics.com>> écrivait: >> >>> Only the first 20 lines are included. There are currently 250000+ >>> more lines with "directory flags set on non-directory inode" and the >>> check is still running (the mostly small files take up around 30 TB, >>> so it'll probably take a while). >>> >>> I recently enabled user and project quota and updated from 3.13.0-53. >>> The file system has been heavily used for the last month or so. >>> >>> Does anyone have any thoughts on this? I'm tempted to stop using >>> quotas when the file system (hopefully) works again, as it's my >>> impression that project quotas are not widely used. >> >> Did you first try remounting then unmounting the volume to clear the >> log? That could clear out xfs_repair output. >> >> -- >> ------------------------------------------------------------------------ >> Emmanuel Florac | Direction technique >> | Intellique >> | <eflorac@intellique.com <mailto:eflorac@intellique.com>> >> | +33 1 78 94 84 02 >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> xfs mailing list >> xfs@oss.sgi.com <mailto:xfs@oss.sgi.com> >> http://oss.sgi.com/mailman/listinfo/xfs > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs [-- Attachment #1.2: Type: text/html, Size: 17554 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-02 7:58 ` Rasmus Borup Hansen @ 2015-07-02 9:26 ` Emmanuel Florac 2015-07-03 6:27 ` Rasmus Borup Hansen 0 siblings, 1 reply; 15+ messages in thread From: Emmanuel Florac @ 2015-07-02 9:26 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs Le Thu, 2 Jul 2015 09:58:19 +0200 Rasmus Borup Hansen <rbh@intomics.com> écrivait: > The file then turns up in lost+found and when I "ls" it I get the > same errors again. I've tried deleting it from lost+found, but then > xfs_repair finds it again with exactly the same output as show above > and puts it back. > > Apart from that, everything apparently works fine. > > Is there a way to permanently get rid of the file in lost+found? Its > size is apparently 0 bytes. You should try the latest xfs_repair (3.2.3 IIRC) instead of the stock Ubuntu version (3.1.x probably). In case you don't know how to compile it, I've just uploaded it there: http://update.intellique.com/pub/xfs_repair-3.2.3.gz md5sum : 756a28228c7e657ce8626d27850f6261 xfs_repair-3.2.3.gz Beware of binaries provided by strangers, however this one should be fine :) gunzip it before use... -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-02 9:26 ` Emmanuel Florac @ 2015-07-03 6:27 ` Rasmus Borup Hansen 2015-07-03 15:24 ` Emmanuel Florac 2015-07-03 23:55 ` Dave Chinner 0 siblings, 2 replies; 15+ messages in thread From: Rasmus Borup Hansen @ 2015-07-03 6:27 UTC (permalink / raw) To: xfs [-- Attachment #1.1: Type: text/plain, Size: 2286 bytes --] Thank you for the suggestion. I compiled a new xfs_repair, but I got similar results from running it. Best, Rasmus Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. ----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark E: rbh@intomics.com W: http://www.intomics.com/ P: +45 5167 7972 P: +45 8880 7979 > On 02 Jul 2015, at 11:26, Emmanuel Florac <eflorac@intellique.com> wrote: > > Le Thu, 2 Jul 2015 09:58:19 +0200 > Rasmus Borup Hansen <rbh@intomics.com> écrivait: > >> The file then turns up in lost+found and when I "ls" it I get the >> same errors again. I've tried deleting it from lost+found, but then >> xfs_repair finds it again with exactly the same output as show above >> and puts it back. >> >> Apart from that, everything apparently works fine. >> >> Is there a way to permanently get rid of the file in lost+found? Its >> size is apparently 0 bytes. > > You should try the latest xfs_repair (3.2.3 IIRC) instead of the stock > Ubuntu version (3.1.x probably). > > In case you don't know how to compile it, I've just uploaded it there: > > http://update.intellique.com/pub/xfs_repair-3.2.3.gz > > md5sum : > 756a28228c7e657ce8626d27850f6261 xfs_repair-3.2.3.gz > > Beware of binaries provided by strangers, however this one should be > fine :) gunzip it before use... > > -- > ------------------------------------------------------------------------ > Emmanuel Florac | Direction technique > | Intellique > | <eflorac@intellique.com> > | +33 1 78 94 84 02 > ------------------------------------------------------------------------ > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs [-- Attachment #1.2: Type: text/html, Size: 5919 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-03 6:27 ` Rasmus Borup Hansen @ 2015-07-03 15:24 ` Emmanuel Florac 2015-07-03 23:55 ` Dave Chinner 1 sibling, 0 replies; 15+ messages in thread From: Emmanuel Florac @ 2015-07-03 15:24 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs Le Fri, 3 Jul 2015 08:27:28 +0200 Rasmus Borup Hansen <rbh@intomics.com> écrivait: > Thank you for the suggestion. I compiled a new xfs_repair, but I got > similar results from running it. > Get the inode number of the faulty file, umount the volume, open it with xfs_db and remove the file through it: http://xfs.org/index.php/XFS_FAQ#Q:_How_to_get_around_a_bad_inode_repair_is_unable_to_clean_up -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-03 6:27 ` Rasmus Borup Hansen 2015-07-03 15:24 ` Emmanuel Florac @ 2015-07-03 23:55 ` Dave Chinner 2015-07-06 11:08 ` Rasmus Borup Hansen 1 sibling, 1 reply; 15+ messages in thread From: Dave Chinner @ 2015-07-03 23:55 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs On Fri, Jul 03, 2015 at 08:27:28AM +0200, Rasmus Borup Hansen wrote: > Thank you for the suggestion. I compiled a new xfs_repair, but I got similar results from running it. Please take a metadump of the filesystem before and after repair so that we can find out what is actually going wrong. Feel free to send me a private pointer to the metadump images... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-03 23:55 ` Dave Chinner @ 2015-07-06 11:08 ` Rasmus Borup Hansen 2015-07-07 0:19 ` Dave Chinner 0 siblings, 1 reply; 15+ messages in thread From: Rasmus Borup Hansen @ 2015-07-06 11:08 UTC (permalink / raw) To: xfs [-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --] I've made a metadump and I'm running another xfs_repair, but given that the first metadump is 132 GB, will you still be interested in looking at the dumps? Best, Rasmus Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. ----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark E: rbh@intomics.com W: http://www.intomics.com/ P: +45 5167 7972 P: +45 8880 7979 > On 04 Jul 2015, at 01:55, Dave Chinner <david@fromorbit.com> wrote: > > On Fri, Jul 03, 2015 at 08:27:28AM +0200, Rasmus Borup Hansen wrote: >> Thank you for the suggestion. I compiled a new xfs_repair, but I got similar results from running it. > > Please take a metadump of the filesystem before and after repair > so that we can find out what is actually going wrong. Feel free to > send me a private pointer to the metadump images... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs [-- Attachment #1.2: Type: text/html, Size: 4479 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-06 11:08 ` Rasmus Borup Hansen @ 2015-07-07 0:19 ` Dave Chinner 2015-07-08 10:28 ` Rasmus Borup Hansen 0 siblings, 1 reply; 15+ messages in thread From: Dave Chinner @ 2015-07-07 0:19 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs On Mon, Jul 06, 2015 at 01:08:52PM +0200, Rasmus Borup Hansen wrote: > I've made a metadump and I'm running another xfs_repair, but given > that the first metadump is 132 GB, will you still be interested in > looking at the dumps? That's significantly larger than my monthly download quota. How big is it once you compress it? Also, because of the size of the metadump, I'll need some context about the hardware it is running on. Can you you please also provide the information in: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F so I have a better idea of environment the problem is showing up in. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-07 0:19 ` Dave Chinner @ 2015-07-08 10:28 ` Rasmus Borup Hansen 2015-07-09 1:15 ` Dave Chinner 0 siblings, 1 reply; 15+ messages in thread From: Rasmus Borup Hansen @ 2015-07-08 10:28 UTC (permalink / raw) To: xfs [-- Attachment #1.1: Type: text/plain, Size: 13452 bytes --] > On 07 Jul 2015, at 02:19, Dave Chinner <david@fromorbit.com> wrote: > > On Mon, Jul 06, 2015 at 01:08:52PM +0200, Rasmus Borup Hansen wrote: >> I've made a metadump and I'm running another xfs_repair, but given >> that the first metadump is 132 GB, will you still be interested in >> looking at the dumps? > > That's significantly larger than my monthly download quota. How big > is it once you compress it? One metadump is 25 GB when compressed with xz -9. The server the files currently reside on is not very fast, so I've only compressed one of them so far. I used the strings command on the metadump files and discovered that they contain fragments of files that we really don't want to leave our IT systems. However, if you think it's worth the effort, I could set up a virtual machine with the metadump files and give you access with your SSH public key. But then you'll have to tell me which tools you'll need for investigating the files. > Also, because of the size of the metadump, I'll need some context > about the hardware it is running on. Can you you please also provide > the information in: > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F > > so I have a better idea of environment the problem is showing up in. $ uname -a Linux mammuthus 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux $ ./xfs_repair -V xfs_repair version 3.2.3 $ cat /proc/cpuinfo | grep processor | wc -l 2 $ cat /proc/meminfo MemTotal: 10228560 kB MemFree: 300612 kB Buffers: 111996 kB Cached: 3569448 kB SwapCached: 7836 kB Active: 1915848 kB Inactive: 2510244 kB Active(anon): 358064 kB Inactive(anon): 386784 kB Active(file): 1557784 kB Inactive(file): 2123460 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2088956 kB SwapFree: 2042964 kB Dirty: 3972 kB Writeback: 0 kB AnonPages: 739808 kB Mapped: 18340 kB Shmem: 72 kB Slab: 4672980 kB SReclaimable: 3927540 kB SUnreclaim: 745440 kB KernelStack: 2328 kB PageTables: 8840 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7203236 kB Committed_AS: 1152936 kB VmallocTotal: 34359738367 kB VmallocUsed: 309048 kB VmallocChunk: 34359412736 kB HardwareCorrupted: 0 kB AnonHugePages: 8192 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 62052 kB DirectMap2M: 10414080 kB $ cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=5103100k,nr_inodes=1275775,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=1022856k,mode=755 0 0 /dev/mapper/mammuthus-root / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 none /sys/fs/pstore pstore rw,relatime 0 0 /dev/sdd1 /boot ext2 rw,relatime 0 0 systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0 /dev/mapper/backup02-limited_backup /limited_backup xfs rw,noatime,attr2,inode64,logbsize=256k,noquota 0 0 /dev/mapper/backup02-timemachine /timemachine_backup ext4 rw,relatime,errors=remount-ro,data=ordered,jqfmt=vfsv0,usrjquota=.aquota.user,grpjquota=.aquota.group 0 0 /dev/mapper/backup01-data /backup xfs rw,noatime,attr2,inode64,logbsize=256k,noquota 0 0 This is what /proc/mounts looks like currently. When the error first occurred there was no /dev/mapper/backup02 volume group and /dev/mapper/backup01-data (which has the file system that behaves strangely) was mounted with user and project quota. Of course, the file system was not mounted when running xfs_repair. $ cat /proc/partitions major minor #blocks name 8 16 19529728000 sdb 8 17 19529726959 sdb1 8 0 39064698880 sda 8 1 39064697839 sda1 252 0 39064694784 dm-0 8 32 976224256 sdc 8 33 976223215 sdc1 8 48 155713536 sdd 8 49 248832 sdd1 8 50 1 sdd2 8 53 155461632 sdd5 252 1 153370624 dm-1 252 2 2088960 dm-2 252 3 33554432 dm-3 252 4 942665728 dm-4 252 5 5368709120 dm-5 252 6 14161014784 dm-6 I'm using hardware RAID level 6 with a Dell PERC H800 controller and an MD1200 disk enclosure with 12 4 TB disks configured as a single "virtual disk": $ /opt/dell/srvadmin/bin/omreport storage vdisk controller=0 List of Virtual Disks on Controller PERC H800 Adapter (Slot 1) Controller PERC H800 Adapter (Slot 1) ID : 0 Status : Ok Name : mammuthus01 State : Ready Hot Spare Policy violated : Not Assigned Encrypted : No Layout : RAID-6 Size : 37,255.00 GB (40002251653120 bytes) T10 Protection Information Status : No Associated Fluid Cache State : Not Applicable Device Name : /dev/sda Bus Protocol : SAS Media : HDD Read Policy : Adaptive Read Ahead Write Policy : Write Back Cache Policy : Not Applicable Stripe Element Size : 64 KB Disk Cache Policy : Disabled This "virtual disk" is the only member of the volume group "backup01": $ sudo pvscan PV /dev/sdd5 VG mammuthus lvm2 [148.26 GiB / 0 free] PV /dev/sdc1 VG extra lvm2 [931.00 GiB / 0 free] PV /dev/sdb1 VG backup02 lvm2 [18.19 TiB / 0 free] PV /dev/sda1 VG backup01 lvm2 [36.38 TiB / 0 free] Total: 4 [55.62 TiB] / in use: 4 [55.62 TiB] / in no VG: 0 [0 ] This volume group has a single logical volume: $ sudo lvscan ACTIVE '/dev/mammuthus/root' [146.27 GiB] inherit ACTIVE '/dev/mammuthus/swap_1' [1.99 GiB] inherit ACTIVE '/dev/extra/swap' [32.00 GiB] inherit ACTIVE '/dev/extra/files' [899.00 GiB] inherit ACTIVE '/dev/backup02/timemachine' [5.00 TiB] inherit ACTIVE '/dev/backup02/limited_backup' [13.19 TiB] inherit ACTIVE '/dev/backup01/data' [36.38 TiB] inherit The drives are 12 4 TB 7.2 RPK Near-Line SAS 3.5" hot plug drives: $ /opt/dell/srvadmin/bin/omreport storage pdisk controller=0 connector=0 List of Physical Disks on Connector 0 of Controller PERC H800 Adapter (Slot 1) Controller PERC H800 Adapter (Slot 1) ID : 0:0:0 Status : Ok Name : Physical Disk 0:0:0 State : Online Power Status : Spun Up Bus Protocol : SAS Media : HDD Part of Cache Pool : Not Applicable Remaining Rated Write Endurance : Not Applicable Failure Predicted : No Revision : GS0D Driver Version : Not Applicable Model Number : Not Applicable T10 PI Capable : No Certified : Yes Encryption Capable : No Encrypted : Not Applicable Progress : Not Applicable Mirror Set ID : Not Applicable Capacity : 3,725.50 GB (4000225165312 bytes) Used RAID Disk Space : 3,725.50 GB (4000225165312 bytes) Available RAID Disk Space : 0.00 GB (0 bytes) Hot Spare : No Vendor ID : DELL(tm) Product ID : ST4000NM0023 Serial No. : Z1Z4BNJ3 Part Number : TH0529FG2123345Q022HA02 Negotiated Speed : 6.00 Gbps Capable Speed : 6.00 Gbps PCIe Maximum Link Width : Not Applicable PCIe Negotiated Link Width : Not Applicable Sector Size : 512B Device Write Cache : Not Applicable Manufacture Day : 04 Manufacture Week : 21 Manufacture Year : 2014 SAS Address : 5000C50058C0F211 (Only output for the first drive show; the others are similar.) The individual drives don't use write caches, but the storage controller has 512 MB cache with battery backup operating in write-back mode. $ xfs_info /backup/ meta-data=/dev/mapper/backup01-data isize=256 agcount=37, agsize=268435455 blks = sectsz=512 attr=2 data = bsize=4096 blocks=9766173696, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Output from ls when listing "lost+found": $ ls -laF /backup/lost+found/ ls: /backup/lost+found/11539619467: Structure needs cleaning total 4 drwxr-xr-x 2 root root 32 Jun 30 07:43 ./ drwxr-xr-x 5 root root 74 Jul 2 12:55 ../ -rw-rw-rw- 1 tsj intomics 0 Jun 23 16:11 11539619467 Relevant output from dmesg (the errors are generated by the ls command above): [444852.252110] XFS (dm-0): Mounting Filesystem [444854.630181] XFS (dm-0): Ending clean mount [503166.397439] ffff880114063000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ [503166.425056] ffff880114063010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... [503166.453484] ffff880114063020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [503166.480812] ffff880114063030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [503166.508386] XFS (dm-0): Internal error xfs_attr3_leaf_read_verify at line 246 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00d4885 [503166.562354] CPU: 1 PID: 3342 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #94-Ubuntu [503166.562356] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011 [503166.562394] Workqueue: xfslogd xfs_buf_iodone_work [xfs] [503166.562398] 0000000000000001 ffff8802af92bd68 ffffffff81723294 ffff88004d1b2000 [503166.562402] ffff8802af92bd80 ffffffffa00d76fb ffffffffa00d4885 ffff8802af92bdb8 [503166.562403] ffffffffa00d7755 000000f600200400 ffff88001b216a00 ffff88004d1b2000 [503166.562406] Call Trace: [503166.562415] [<ffffffff81723294>] dump_stack+0x45/0x56 [503166.562426] [<ffffffffa00d76fb>] xfs_error_report+0x3b/0x40 [xfs] [503166.562436] [<ffffffffa00d4885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [503166.562446] [<ffffffffa00d7755>] xfs_corruption_error+0x55/0x80 [xfs] [503166.562459] [<ffffffffa00f4bdd>] xfs_attr3_leaf_read_verify+0x6d/0xf0 [xfs] [503166.562469] [<ffffffffa00d4885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [503166.562479] [<ffffffffa00d4885>] xfs_buf_iodone_work+0x85/0xf0 [xfs] [503166.562483] [<ffffffff81083b22>] process_one_work+0x182/0x450 [503166.562485] [<ffffffff81084911>] worker_thread+0x121/0x410 [503166.562487] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430 [503166.562489] [<ffffffff8108b702>] kthread+0xd2/0xf0 [503166.562491] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [503166.562494] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90 [503166.562496] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [503166.562498] XFS (dm-0): Corruption detected. Unmount and run xfs_repair [503166.589297] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8 The error occurs even though the file system is not doing anything else. Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data. ----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark E: rbh@intomics.com W: http://www.intomics.com/ P: +45 5167 7972 P: +45 8880 7979 [-- Attachment #1.2: Type: text/html, Size: 35449 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-08 10:28 ` Rasmus Borup Hansen @ 2015-07-09 1:15 ` Dave Chinner 2015-11-13 6:39 ` Arkadiusz Bubała 0 siblings, 1 reply; 15+ messages in thread From: Dave Chinner @ 2015-07-09 1:15 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs On Wed, Jul 08, 2015 at 12:28:19PM +0200, Rasmus Borup Hansen wrote: > > On 07 Jul 2015, at 02:19, Dave Chinner <david@fromorbit.com> > > wrote: > > > > On Mon, Jul 06, 2015 at 01:08:52PM +0200, Rasmus Borup Hansen > > wrote: > >> I've made a metadump and I'm running another xfs_repair, but > >> given that the first metadump is 132 GB, will you still be > >> interested in looking at the dumps? > > > > That's significantly larger than my monthly download quota. How > > big is it once you compress it? > > One metadump is 25 GB when compressed with xz -9. The server the > files currently reside on is not very fast, so I've only > compressed one of them so far. > > I used the strings command on the metadump files and discovered > that they contain fragments of files that we really don't want to > leave our IT systems. However, if you think it's worth the effort, > I could set up a virtual machine with the metadump files and give > you access with your SSH public key. But then you'll have to tell > me which tools you'll need for investigating the files. <sigh> I only want to look at one inode with xfs_db.... I don't think ssh access is going be very useful. To isolate the actual cause of the verifier error I usually run an instrumented kernel, and to verify that I've fixed the problem I'll need to run custom kernels and/or xfsprogs binaries. So I really need a local fs image to do this. I also like to run a kernel I trust (i.e. one I've built myself) on a machine I trust not to have storage or hardware issues when doing diagnostics on broken filesystem images. Of course, nobody can learn about how to find problems like this when the triage is hidden away in private.... > Output from ls when listing "lost+found": > > $ ls -laF /backup/lost+found/ > ls: /backup/lost+found/11539619467: Structure needs cleaning This is the inode I need to look at. Can you grep for this inode in the xfs_repair output and post it? (grab a couple of lines of context around each match, too...) > total 4 > drwxr-xr-x 2 root root 32 Jun 30 07:43 ./ > drwxr-xr-x 5 root root 74 Jul 2 12:55 ../ > -rw-rw-rw- 1 tsj intomics 0 Jun 23 16:11 11539619467 It's a zero length file, so I'll be interested to know what attributes it has on it. Can you check that the inode number of that file is 11539619467 (ls -i will tell you that) and then post the output of this command: # xfs_db -r -c "inode 11539619467" -c p /dev/mapper/backup01-data Then I can give you the commands for walking and dumping the full attribute tree. > [503166.562498] XFS (dm-0): Corruption detected. Unmount and run xfs_repair > [503166.589297] XFS (dm-0): metadata I/O error: block 0x157e84da0 ("xfs_trans_read_buf_map") error 117 numblks 8 Also, post the output of: # xfs_db -r -c "convert daddr 0x157e84da0 fsb" /dev/mapper/backup01-data <fsb> # xfs_db -r -c "fsb <fsb>" -c p -c "type attr" -c p /dev/mapper/backup01-data Which will give me the contents of the bad block, both in raw format and as processed by the attribute leaf format parser in xfs_db. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-07-09 1:15 ` Dave Chinner @ 2015-11-13 6:39 ` Arkadiusz Bubała 0 siblings, 0 replies; 15+ messages in thread From: Arkadiusz Bubała @ 2015-11-13 6:39 UTC (permalink / raw) To: xfs Hello, I encountered the same issue on the Linux kernel 3.10. Here's the information you previously asked about. I hope it'll help to find the root of this problem. The ls command invoked on lost+found: #ls -la ls: 246185005270: Operation not permitted total 4 drwxr-xr-x 2 root root 33 Nov 7 00:51 . drwxr-xr-x 5 root root 61 Sep 21 21:07 .. -rwxrwxrwx 1 user user 0 Oct 27 18:22 246185005270 Execution of ls generates following error in the dmesg: [486147.297493] ffff88063af38000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ [486147.318762] ffff88063af38010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... [486147.339328] ffff88063af38020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [486147.359597] ffff88063af38030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [486147.359599] XFS (dm-16): Internal error xfs_attr3_leaf_read_verify at line 256 of file fs/xfs/xfs_attr_leaf.c. Caller 0xffffffff81263bcd [486147.359599] [486147.359601] CPU: 8 PID: 14315 Comm: kworker/8:1H Tainted: G O 3.10.80 #45 [486147.359602] Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 [486147.359607] Workqueue: xfslogd xfs_buf_iodone_work [486147.359611] ffffffff816bf9df 000000000000007e ffffffff81265a42 ffffffff81263bcd [486147.359612] ffff000000000100 ffff8800518b4fc0 ffff88000de01200 ffff88085adf5000 [486147.359613] ffff88087fd18f00 0000000000000000 ffffffff8127e95f ffffffff81263bcd [486147.359614] Call Trace: [486147.359617] [<ffffffff816bf9df>] ? dump_stack+0xd/0x17 [486147.359620] [<ffffffff81265a42>] ? xfs_corruption_error+0x62/0x90 [486147.359621] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0 [486147.359624] [<ffffffff8127e95f>] ? xfs_attr3_leaf_read_verify+0x6f/0x100 [486147.359625] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0 [486147.359627] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0 [486147.359630] [<ffffffff8105dd01>] ? process_one_work+0x141/0x3b0 [486147.359633] [<ffffffff8105b79e>] ? pwq_activate_delayed_work+0x2e/0x50 [486147.359636] [<ffffffff8105ec42>] ? worker_thread+0x112/0x370 [486147.359638] [<ffffffff8105eb30>] ? manage_workers.isra.28+0x290/0x290 [486147.359641] [<ffffffff810647e3>] ? kthread+0xb3/0xc0 [486147.359644] [<ffffffff81012685>] ? sched_clock+0x5/0x10 [486147.359646] [<ffffffff81060000>] ? __alloc_workqueue_key+0x210/0x510 [486147.359648] [<ffffffff81064730>] ? kthread_freezable_should_stop+0x60/0x60 [486147.359652] [<ffffffff816cab18>] ? ret_from_fork+0x58/0x90 [486147.359654] [<ffffffff81064730>] ? kthread_freezable_should_stop+0x60/0x60 [486147.359655] XFS (dm-16): Corruption detected. Unmount and run xfs_repair [486147.359664] XFS (dm-16): metadata I/O error: block 0x1ca8ebe310 ("xfs_trans_read_buf_map") error 117 numblks 8 # xfs_db -r /dev/mapper/vg+vg01-lv+n+lv0100 xfs_db> inode 246185005270 xfs_db> p core.magic = 0x494e core.mode = 0100777 core.version = 2 core.format = 2 (extents) core.nlinkv2 = 1 core.onlink = 0 core.projid_lo = 0 core.projid_hi = 0 core.uid = 106 core.gid = 103 core.flushiter = 4 core.atime.sec = Tue Oct 27 18:22:48 2015 core.atime.nsec = 000000000 core.mtime.sec = Tue Oct 27 18:22:48 2015 core.mtime.nsec = 000000000 core.ctime.sec = Sun Nov 1 19:05:21 2015 core.ctime.nsec = 337962467 core.size = 0 core.nblocks = 1 core.extsize = 0 core.nextents = 0 core.naextents = 1 core.forkoff = 9 core.aformat = 2 (extents) core.dmevmask = 0 core.dmstate = 0 core.newrtbm = 0 core.prealloc = 0 core.realtime = 0 core.immutable = 0 core.append = 0 core.sync = 0 core.noatime = 0 core.nodump = 0 core.rtinherit = 0 core.projinherit = 0 core.nosymlinks = 0 core.extsz = 0 core.extszinherit = 0 core.nodefrag = 0 core.filestream = 0 core.gen = 2298109809 next_unlinked = null u = (empty) a.bmx[0] = [startoff,startblock,blockcount,extentflag] 0:[0,15386639515,1,0] xfs_db> ablock 0 xfs_db> p hdr.info.forw = 0 hdr.info.back = 0 hdr.info.magic = 0xfbee hdr.count = 0 hdr.usedbytes = 0 hdr.firstused = 4096 hdr.holes = 0 hdr.freemap[0-2] = [base,size] 0:[32,4064] 1:[0,0] 2:[0,0] xfs_db> type text xfs_db> p 000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ 010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ................ 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ ... (zeros only) ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ #xfs_db -r -c "convert daddr 0x1ca8ebe310 fsb" /dev/mapper/lv0100 0x3951d7c9b (15386639515) # xfs_db -r -c "fsb 0x3951d7c9b" -c p -c "type attr" -c p /dev/mapper/lv0100 000: 00000000 00000000 fbee0000 00000000 10000000 00200fe0 00000000 00000000 020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 040: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 060: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 080: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 100: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 120: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 140: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 180: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 1a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 1c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 1e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 260: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 300: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 340: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 380: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 620: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 640: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 660: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 680: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 700: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 720: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 740: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 760: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 780: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 7a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 7c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 7e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 800: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 820: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 840: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 860: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 880: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 8a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 8c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 8e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 900: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 920: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 940: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 960: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 980: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 9a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 9c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 9e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 a80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 aa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ac0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ae0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ba0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 bc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 be0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 c80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ca0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 cc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ce0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 d00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 d20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 d40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 d60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 d80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 da0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 de0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 e00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 e20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 e40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 e60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 e80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 fe0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 hdr.info.forw = 0 hdr.info.back = 0 hdr.info.magic = 0xfbee hdr.count = 0 hdr.usedbytes = 0 hdr.firstused = 4096 hdr.holes = 0 hdr.freemap[0-2] = [base,size] 0:[32,4064] 1:[0,0] 2:[0,0] -- Best regards Arkadiusz Bubała Open-E Poland Sp. z o.o. www.open-e.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors 2015-06-24 7:39 "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors Rasmus Borup Hansen 2015-06-25 16:41 ` Emmanuel Florac @ 2015-06-29 21:50 ` Dave Chinner 1 sibling, 0 replies; 15+ messages in thread From: Dave Chinner @ 2015-06-29 21:50 UTC (permalink / raw) To: Rasmus Borup Hansen; +Cc: xfs On Wed, Jun 24, 2015 at 09:39:45AM +0200, Rasmus Borup Hansen wrote: > Hi! Yesterday I got the following error messages from the kernel (Ubuntu trusty, 3.13.0-55): > > [601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ > [601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... > [601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [601201.820013] XFS (dm-0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00996f0 Probably fixed by commit c88547a ("xfs: fix directory hash ordering bug"), which also affected attributes in leaf format. > I'm currently running xfs_repair -n and so far I've seen the following output: > > Phase 1 - find and verify superblock... > Phase 2 - using internal log > - scan filesystem freespace and inode maps... > - found root inode chunk > Phase 3 - for each AG... > - scan (but don't clear) agi unlinked lists... > - process known inodes and perform inode discovery... > - agno = 0 > doubling cache size to 591200 > directory flags set on non-directory inode 206624 > directory flags set on non-directory inode 206625 > directory flags set on non-directory inode 206626 > directory flags set on non-directory inode 206627 > directory flags set on non-directory inode 206628 > directory flags set on non-directory inode 206629 > directory flags set on non-directory inode 206630 > directory flags set on non-directory inode 206631 > directory flags set on non-directory inode 206632 > directory flags set on non-directory inode 206633 > directory flags set on non-directory inode 206634 > > Only the first 20 lines are included. There are currently 250000+ > more lines with "directory flags set on non-directory inode" and > the check is still running (the mostly small files take up around > 30 TB, so it'll probably take a while). Harmless, but repair will fix it anyway. Kernel is fixed by commit 9336e3a ("xfs: project id inheritance is a directory only flag"). Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors
@ 2015-11-19 13:03 Adam Błaszczykowski
0 siblings, 0 replies; 15+ messages in thread
From: Adam Błaszczykowski @ 2015-11-19 13:03 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 15618 bytes --]
Hello,
I have updated kernel version 3.4 to 3.10 and I encountered the same
problem. System is working, but periodically I get following errors in
kernel logs:
[20367.539731] XFS (dm-4): Internal error xfs_attr3_leaf_read_verify at
line 256 of file fs/xfs/xfs_attr_leaf.c. Caller 0xffffffff81263bcd
[20367.539731]
[20367.539732] CPU: 1 PID: 2240 Comm: kworker/1:1H Tainted: G O
3.10.93 #15
[20367.539733] Hardware name: Intel Corporation S2600IP/S2600IP, BIOS
SE5C600.86B.02.03.0003.041920141333 04/19/2014
[20367.539736] Workqueue: xfslogd xfs_buf_iodone_work
[20367.539738] ffffffff816bf9df 000000000000007e ffffffff81265a42
ffffffff81263bcd
[20367.539739] ffff000000000100 ffff8802c8e40e40 ffff88085257f080
ffff88085adf5000
[20367.539740] ffff88087fc38f00 0000000000000000 ffffffff8127e95f
ffffffff81263bcd
[20367.539741] Call Trace:
[20367.539744] [<ffffffff816bf9df>] ? dump_stack+0xd/0x17
[20367.539746] [<ffffffff81265a42>] ? xfs_corruption_error+0x62/0x90
[20367.539748] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539750] [<ffffffff8127e95f>] ? xfs_attr3_leaf_read_verify+0x6f/0x100
[20367.539751] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539753] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539754] [<ffffffff8105dd01>] ? process_one_work+0x141/0x3b0
[20367.539757] [<ffffffff8105b79e>] ? pwq_activate_delayed_work+0x2e/0x50
[20367.539758] [<ffffffff8105ec42>] ? worker_thread+0x112/0x370
[20367.539759] [<ffffffff8105eb30>] ? manage_workers.isra.28+0x290/0x290
[20367.539761] [<ffffffff810647e3>] ? kthread+0xb3/0xc0
[20367.539763] [<ffffffff81012685>] ? sched_clock+0x5/0x10
[20367.539764] [<ffffffff81060000>] ? __alloc_workqueue_key+0x210/0x510
[20367.539766] [<ffffffff81064730>] ?
kthread_freezable_should_stop+0x60/0x60
[20367.539768] [<ffffffff816cab18>] ? ret_from_fork+0x58/0x90
[20367.539770] [<ffffffff81064730>] ?
kthread_freezable_should_stop+0x60/0x60
[20367.539770] XFS (dm-4): Corruption detected. Unmount and run xfs_repair
[20367.539777] XFS (dm-4): metadata I/O error: block 0x19a4775b18
("xfs_trans_read_buf_map") error 117 numblks 8
As in other cases from this thread, xfs repair hasn't found any problems.
Here is xfs_dump from block 0x19a4775b18:
[20367.539731] XFS (dm-4): Internal error xfs_attr3_leaf_read_verify at
line 256 of file fs/xfs/xfs_attr_leaf.c. Caller 0xffffffff81263bcd
[20367.539731]
[20367.539732] CPU: 1 PID: 2240 Comm: kworker/1:1H Tainted: G O
3.10.93 #15
[20367.539733] Hardware name: Intel Corporation S2600IP/S2600IP, BIOS
SE5C600.86B.02.03.0003.041920141333 04/19/2014
[20367.539736] Workqueue: xfslogd xfs_buf_iodone_work
[20367.539738] ffffffff816bf9df 000000000000007e ffffffff81265a42
ffffffff81263bcd
[20367.539739] ffff000000000100 ffff8802c8e40e40 ffff88085257f080
ffff88085adf5000
[20367.539740] ffff88087fc38f00 0000000000000000 ffffffff8127e95f
ffffffff81263bcd
[20367.539741] Call Trace:
[20367.539744] [<ffffffff816bf9df>] ? dump_stack+0xd/0x17
[20367.539746] [<ffffffff81265a42>] ? xfs_corruption_error+0x62/0x90
[20367.539748] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539750] [<ffffffff8127e95f>] ? xfs_attr3_leaf_read_verify+0x6f/0x100
[20367.539751] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539753] [<ffffffff81263bcd>] ? xfs_buf_iodone_work+0x8d/0xb0
[20367.539754] [<ffffffff8105dd01>] ? process_one_work+0x141/0x3b0
[20367.539757] [<ffffffff8105b79e>] ? pwq_activate_delayed_work+0x2e/0x50
[20367.539758] [<ffffffff8105ec42>] ? worker_thread+0x112/0x370
[20367.539759] [<ffffffff8105eb30>] ? manage_workers.isra.28+0x290/0x290
[20367.539761] [<ffffffff810647e3>] ? kthread+0xb3/0xc0
[20367.539763] [<ffffffff81012685>] ? sched_clock+0x5/0x10
[20367.539764] [<ffffffff81060000>] ? __alloc_workqueue_key+0x210/0x510
[20367.539766] [<ffffffff81064730>] ?
kthread_freezable_should_stop+0x60/0x60
[20367.539768] [<ffffffff816cab18>] ? ret_from_fork+0x58/0x90
[20367.539770] [<ffffffff81064730>] ?
kthread_freezable_should_stop+0x60/0x60
[20367.539770] XFS (dm-4): Corruption detected. Unmount and run xfs_repair
[20367.539777] XFS (dm-4): metadata I/O error: block 0x19a4775b18
("xfs_trans_read_buf_map") error 117 numblks 8
xfs_db> convert daddr 0x19a4775b18 fsb
0x3348eeb96 (13766683542)
xfs_db> fsb 0x3348eeb96
xfs_db> p
000: 00000000 00000000 fbee0000 00000000 10000000 00200fe0 00000000 00000000
020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
040: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
060: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
080: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
100: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
120: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
140: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
160: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
180: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
260: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
2e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
300: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
340: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
380: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
3e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
620: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
640: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
660: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
680: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
6e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
700: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
720: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
740: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
760: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
780: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
800: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
820: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
840: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
860: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
880: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
8a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
8c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
8e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
900: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
920: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
940: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
960: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
980: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
9a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
9c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
9e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
a00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
a20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
a40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
a60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
a80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
aa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ac0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ae0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
b80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ba0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
be0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
c80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ca0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
cc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ce0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
d00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
d20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
d40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
d60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
d80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
da0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
dc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
de0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
e00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
e20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
e40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
e60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
e80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
fe0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
xfs_db> type attr
xfs_db> p
hdr.info.forw = 0
hdr.info.back = 0
hdr.info.magic = 0xfbee
hdr.count = 0
hdr.usedbytes = 0
hdr.firstused = 4096
hdr.holes = 0
hdr.freemap[0-2] = [base,size] 0:[32,4064] 1:[0,0] 2:[0,0]
After converting the address to inode and running xfs_dump I get following
output:
xfs_db> convert daddr 0x19a4775b18 inode
0x3348eeb960 (220266936672)
xfs_db> inode 0x3348eeb960
xfs_db> p
core.magic = 0
core.mode = 0
core.version = 0
core.format = 0 (dev)
core.uid = 4226678784
core.gid = 0
core.flushiter = 0
core.atime.sec = Thu Jan 1 01:00:00 1970
core.atime.nsec = 000000000
core.mtime.sec = Thu Jan 1 01:00:00 1970
core.mtime.nsec = 000000000
core.ctime.sec = Thu Jan 1 01:00:00 1970
core.ctime.nsec = 000000000
core.size = 0
core.nblocks = 0
core.extsize = 0
core.nextents = 0
core.naextents = 0
core.forkoff = 0
core.aformat = 0 (dev)
core.dmevmask = 0
core.dmstate = 0
core.newrtbm = 0
core.prealloc = 0
core.realtime = 0
core.immutable = 0
core.append = 0
core.sync = 0
core.noatime = 0
core.nodump = 0
core.rtinherit = 0
core.projinherit = 0
core.nosymlinks = 0
core.extsz = 0
core.extszinherit = 0
core.nodefrag = 0
core.filestream = 0
core.gen = 0
next_unlinked = 0
u.dev = 0
It's strange, because there is no "extended attributes" record.
Do you know what can cause such problem ?
Is it fixed in new kernel version ?
Thank you in advance !
Best regards
Adam Blaszczykowski
[-- Attachment #1.2: Type: text/html, Size: 16754 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 15+ messages in threadend of thread, other threads:[~2015-11-19 13:03 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-06-24 7:39 "Internal error xfs_attr3_leaf_write_verify at line 216", "directory flags set on non-directory inode" and other errors Rasmus Borup Hansen 2015-06-25 16:41 ` Emmanuel Florac 2015-06-26 6:14 ` Rasmus Borup Hansen 2015-07-02 7:58 ` Rasmus Borup Hansen 2015-07-02 9:26 ` Emmanuel Florac 2015-07-03 6:27 ` Rasmus Borup Hansen 2015-07-03 15:24 ` Emmanuel Florac 2015-07-03 23:55 ` Dave Chinner 2015-07-06 11:08 ` Rasmus Borup Hansen 2015-07-07 0:19 ` Dave Chinner 2015-07-08 10:28 ` Rasmus Borup Hansen 2015-07-09 1:15 ` Dave Chinner 2015-11-13 6:39 ` Arkadiusz Bubała 2015-06-29 21:50 ` Dave Chinner -- strict thread matches above, loose matches on Subject: below -- 2015-11-19 13:03 Adam Błaszczykowski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox