* Structure needs cleaning? (take #2) @ 2009-09-02 13:22 RUMI Szabolcs 2009-09-02 14:31 ` Eric Sandeen 0 siblings, 1 reply; 5+ messages in thread From: RUMI Szabolcs @ 2009-09-02 13:22 UTC (permalink / raw) To: xfs Hi! Sorry but my previous post was missing the important first two lines: d62c8000: 2c 30 78 41 41 41 41 30 30 30 30 2c 36 2c 30 78 ,0xAAAA0000,6,0x Filesystem "sda10": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xc02cc790 Pid: 29510, comm: mc Tainted: P 2.6.29-gentoo-r5-PAE #1 Call Trace: [<c02cc6c4>] xfs_da_do_buf+0x8c4/0x900 [<c02cc790>] xfs_da_read_buf+0x30/0x40 [<c02cc790>] xfs_da_read_buf+0x30/0x40 [<c0191ef0>] pollwake+0x0/0x50 [<c0191ef0>] pollwake+0x0/0x50 [<c02cc790>] xfs_da_read_buf+0x30/0x40 [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 [<c02d3667>] xfs_dir2_leaf_lookup+0x27/0xc0 [<c02cf32f>] xfs_dir2_isleaf+0x1f/0x60 [<c02cfc78>] xfs_dir_lookup+0xd8/0x180 [<c02fedab>] xfs_lookup+0x6b/0xf0 [<c0309165>] xfs_vn_lookup+0x55/0xa0 [<c018adda>] do_lookup+0x1ba/0x1e0 [<c018ca1d>] __link_path_walk+0x6cd/0xd60 [<c02d3def>] xfs_dir2_leaf_getdents+0x5ff/0xad0 [<c018d264>] path_walk+0x54/0xc0 [<c018d3a3>] do_path_lookup+0x83/0x170 [<c018c30b>] getname+0x9b/0xe0 [<c018e03a>] user_path_at+0x5a/0x90 [<c018633f>] vfs_lstat_fd+0x1f/0x50 [<c01863bf>] sys_lstat64+0xf/0x30 [<c0194fd4>] touch_atime+0x14/0x130 [<c01907b8>] vfs_readdir+0x78/0xb0 [<c0190891>] sys_getdents64+0xa1/0xd0 [<c01032f1>] sysenter_do_call+0x12/0x25 Thanks, Sab _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Structure needs cleaning? (take #2) 2009-09-02 13:22 Structure needs cleaning? (take #2) RUMI Szabolcs @ 2009-09-02 14:31 ` Eric Sandeen 2009-09-02 19:34 ` RUMI Szabolcs 0 siblings, 1 reply; 5+ messages in thread From: Eric Sandeen @ 2009-09-02 14:31 UTC (permalink / raw) To: RUMI Szabolcs; +Cc: xfs RUMI Szabolcs wrote: > Hi! > > Sorry but my previous post was missing the important first two lines: Yes, thanks. :) > d62c8000: 2c 30 78 41 41 41 41 30 30 30 30 2c 36 2c 30 78 ,0xAAAA0000,6,0x > Filesystem "sda10": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xc02cc790 This is on-disk corruption, it found bad magic on something it expected to be metadata. You should run xfs_repair. run with -n, or on a restored xfs_metadump image as a dry-run first, if you prefer. -Eric > Pid: 29510, comm: mc Tainted: P 2.6.29-gentoo-r5-PAE #1 > Call Trace: > [<c02cc6c4>] xfs_da_do_buf+0x8c4/0x900 > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > [<c0191ef0>] pollwake+0x0/0x50 > [<c0191ef0>] pollwake+0x0/0x50 > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 > [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 > [<c02d3667>] xfs_dir2_leaf_lookup+0x27/0xc0 > [<c02cf32f>] xfs_dir2_isleaf+0x1f/0x60 > [<c02cfc78>] xfs_dir_lookup+0xd8/0x180 > [<c02fedab>] xfs_lookup+0x6b/0xf0 > [<c0309165>] xfs_vn_lookup+0x55/0xa0 > [<c018adda>] do_lookup+0x1ba/0x1e0 > [<c018ca1d>] __link_path_walk+0x6cd/0xd60 > [<c02d3def>] xfs_dir2_leaf_getdents+0x5ff/0xad0 > [<c018d264>] path_walk+0x54/0xc0 > [<c018d3a3>] do_path_lookup+0x83/0x170 > [<c018c30b>] getname+0x9b/0xe0 > [<c018e03a>] user_path_at+0x5a/0x90 > [<c018633f>] vfs_lstat_fd+0x1f/0x50 > [<c01863bf>] sys_lstat64+0xf/0x30 > [<c0194fd4>] touch_atime+0x14/0x130 > [<c01907b8>] vfs_readdir+0x78/0xb0 > [<c0190891>] sys_getdents64+0xa1/0xd0 > [<c01032f1>] sysenter_do_call+0x12/0x25 > > Thanks, > Sab > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Structure needs cleaning? (take #2) 2009-09-02 14:31 ` Eric Sandeen @ 2009-09-02 19:34 ` RUMI Szabolcs 2009-09-02 19:53 ` Eric Sandeen 0 siblings, 1 reply; 5+ messages in thread From: RUMI Szabolcs @ 2009-09-02 19:34 UTC (permalink / raw) To: Eric Sandeen; +Cc: xfs Hi! Well, what could be the reason? I mean, there was no hardware failure, no crash, no reboot, no errors in the disk's SMART error log, no nothing. What I did was that I've extracted and deleted the rather huge OpenOffice source tree several times (sometimes with overwriting) and finally it ended up with these undeletable files and xfs errors. Is it considered normal for xfs to get messed up like that under such load? Thanks, Sab The xfs_repair output and xfs_info output is included below: # xfs_repair -v /dev/sda10 Phase 1 - find and verify superblock... - block cache size set to 255664 entries Phase 2 - using internal log - zero log... zero_log: head block 37688 tail block 37688 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - agno = 27 - agno = 28 - agno = 29 - agno = 30 - agno = 31 - agno = 32 - agno = 33 - agno = 34 - agno = 35 - agno = 36 - agno = 37 - agno = 38 - agno = 39 - agno = 40 - agno = 41 - agno = 42 - agno = 43 - agno = 44 - agno = 45 - agno = 46 - agno = 47 - agno = 48 - agno = 49 - agno = 50 - agno = 51 - agno = 52 - agno = 53 - agno = 54 - agno = 55 - agno = 56 - agno = 57 - agno = 58 - agno = 59 - agno = 60 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - agno = 27 - agno = 28 - agno = 29 - agno = 30 - agno = 31 - agno = 32 - agno = 33 - agno = 34 - agno = 35 - agno = 36 - agno = 37 - agno = 38 - agno = 39 - agno = 40 - agno = 41 - agno = 42 - agno = 43 - agno = 44 - agno = 45 - agno = 46 - agno = 47 - agno = 48 - agno = 49 - agno = 50 - agno = 51 - agno = 52 - agno = 53 - agno = 54 - agno = 55 - agno = 56 - agno = 57 - agno = 58 - agno = 59 - agno = 60 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - agno = 27 - agno = 28 - agno = 29 - agno = 30 - agno = 31 - agno = 32 - agno = 33 - agno = 34 - agno = 35 - agno = 36 - agno = 37 - agno = 38 - agno = 39 - agno = 40 - agno = 41 - agno = 42 - agno = 43 - agno = 44 - agno = 45 - agno = 46 - agno = 47 - agno = 48 - agno = 49 - agno = 50 - agno = 51 - agno = 52 - agno = 53 - agno = 54 - agno = 55 - agno = 56 - agno = 57 - agno = 58 - agno = 59 - agno = 60 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 leaf block 8388608 for directory inode 4051737 bad header rebuilding directory inode 4051737 leaf block 8388608 for directory inode 4053318 bad header rebuilding directory inode 4053318 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - agno = 16 - agno = 17 - agno = 18 - agno = 19 - agno = 20 - agno = 21 - agno = 22 - agno = 23 - agno = 24 - agno = 25 - agno = 26 - agno = 27 - agno = 28 - agno = 29 - agno = 30 - agno = 31 - agno = 32 - agno = 33 - agno = 34 - agno = 35 - agno = 36 - agno = 37 - agno = 38 - agno = 39 - agno = 40 - agno = 41 - agno = 42 - agno = 43 - agno = 44 - agno = 45 - agno = 46 - agno = 47 - agno = 48 - agno = 49 - agno = 50 - agno = 51 - agno = 52 - agno = 53 - agno = 54 - agno = 55 - agno = 56 - agno = 57 - agno = 58 - agno = 59 - agno = 60 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... XFS_REPAIR Summary Wed Sep 2 21:23:53 2009 Phase Start End Duration Phase 1: 09/02 21:23:33 09/02 21:23:33 Phase 2: 09/02 21:23:33 09/02 21:23:35 2 seconds Phase 3: 09/02 21:23:35 09/02 21:23:49 14 seconds Phase 4: 09/02 21:23:49 09/02 21:23:50 1 second Phase 5: 09/02 21:23:50 09/02 21:23:50 Phase 6: 09/02 21:23:50 09/02 21:23:50 Phase 7: 09/02 21:23:50 09/02 21:23:50 Total run time: 17 seconds done # xfs_info /dev/sda10 meta-data=/dev/sda10 isize=256 agcount=61, agsize=32768 blks = sectsz=512 attr=0 data = bsize=4096 blocks=1998848, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=16384, version=2 = sectsz=512 sunit=1 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 On Wed, 02 Sep 2009 09:31:09 -0500 Eric Sandeen <sandeen@sandeen.net> wrote: > RUMI Szabolcs wrote: > > Hi! > > > > Sorry but my previous post was missing the important first two lines: > > Yes, thanks. :) > > > d62c8000: 2c 30 78 41 41 41 41 30 30 30 30 2c 36 2c 30 78 ,0xAAAA0000,6,0x > > Filesystem "sda10": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xc02cc790 > > This is on-disk corruption, it found bad magic on something it expected > to be metadata. You should run xfs_repair. run with -n, or on a > restored xfs_metadump image as a dry-run first, if you prefer. > > -Eric > > > Pid: 29510, comm: mc Tainted: P 2.6.29-gentoo-r5-PAE #1 > > Call Trace: > > [<c02cc6c4>] xfs_da_do_buf+0x8c4/0x900 > > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > > [<c0191ef0>] pollwake+0x0/0x50 > > [<c0191ef0>] pollwake+0x0/0x50 > > [<c02cc790>] xfs_da_read_buf+0x30/0x40 > > [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 > > [<c02d2943>] xfs_dir2_leaf_lookup_int+0x63/0x2f0 > > [<c02d3667>] xfs_dir2_leaf_lookup+0x27/0xc0 > > [<c02cf32f>] xfs_dir2_isleaf+0x1f/0x60 > > [<c02cfc78>] xfs_dir_lookup+0xd8/0x180 > > [<c02fedab>] xfs_lookup+0x6b/0xf0 > > [<c0309165>] xfs_vn_lookup+0x55/0xa0 > > [<c018adda>] do_lookup+0x1ba/0x1e0 > > [<c018ca1d>] __link_path_walk+0x6cd/0xd60 > > [<c02d3def>] xfs_dir2_leaf_getdents+0x5ff/0xad0 > > [<c018d264>] path_walk+0x54/0xc0 > > [<c018d3a3>] do_path_lookup+0x83/0x170 > > [<c018c30b>] getname+0x9b/0xe0 > > [<c018e03a>] user_path_at+0x5a/0x90 > > [<c018633f>] vfs_lstat_fd+0x1f/0x50 > > [<c01863bf>] sys_lstat64+0xf/0x30 > > [<c0194fd4>] touch_atime+0x14/0x130 > > [<c01907b8>] vfs_readdir+0x78/0xb0 > > [<c0190891>] sys_getdents64+0xa1/0xd0 > > [<c01032f1>] sysenter_do_call+0x12/0x25 > > > > Thanks, > > Sab > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com > > http://oss.sgi.com/mailman/listinfo/xfs > > > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Structure needs cleaning? (take #2) 2009-09-02 19:34 ` RUMI Szabolcs @ 2009-09-02 19:53 ` Eric Sandeen 2009-09-02 21:02 ` Justin Piszcz 0 siblings, 1 reply; 5+ messages in thread From: Eric Sandeen @ 2009-09-02 19:53 UTC (permalink / raw) To: RUMI Szabolcs; +Cc: xfs RUMI Szabolcs wrote: > Hi! > > Well, what could be the reason? I mean, there was no hardware failure, > no crash, no reboot, no errors in the disk's SMART error log, no nothing. > What I did was that I've extracted and deleted the rather huge OpenOffice > source tree several times (sometimes with overwriting) and finally it > ended up with these undeletable files and xfs errors. Is it considered > normal for xfs to get messed up like that under such load? No, not normal. It found the text "0xAAAA0000,6,0x" on disk in an area where it expected to find valid filesystem metadata. Corruption could come from anywhere - an xfs bug, some other bug, bad memory, bad cables, neon death rays from space, writing directly to the disk, who knows. Awfully hard to track down a one-off occurrence like this, I'm afraid. > Thanks, > Sab > > > > The xfs_repair output and xfs_info output is included below: > > # xfs_repair -v /dev/sda10 ... > Phase 6 - check inode connectivity... > - resetting contents of realtime bitmap and summary inodes > - traversing filesystem ... > - agno = 0 > - agno = 1 > - agno = 2 > - agno = 3 > - agno = 4 > - agno = 5 > - agno = 6 > - agno = 7 > leaf block 8388608 for directory inode 4051737 bad header > rebuilding directory inode 4051737 > leaf block 8388608 for directory inode 4053318 bad header > rebuilding directory inode 4053318 ... above is the problem, properly found & repaired. -Eric _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Structure needs cleaning? (take #2) 2009-09-02 19:53 ` Eric Sandeen @ 2009-09-02 21:02 ` Justin Piszcz 0 siblings, 0 replies; 5+ messages in thread From: Justin Piszcz @ 2009-09-02 21:02 UTC (permalink / raw) To: Eric Sandeen; +Cc: RUMI Szabolcs, xfs Hi RUMI, 1. Have you run memtest86 for a few passes? 2. Have you run a short+long test against the drive? smartctl -t short /dev/sda # wait 10min smartctl -t long # wait until its done show smartctl -a /dev/sda output <- report this back 3. Is this the first time this has happened? 4. Is your system on a UPS? 5. How do you mount your XFS partition? a. Do you use any special parameters? Justin. On Wed, 2 Sep 2009, Eric Sandeen wrote: > RUMI Szabolcs wrote: >> Hi! >> >> Well, what could be the reason? I mean, there was no hardware failure, >> no crash, no reboot, no errors in the disk's SMART error log, no nothing. >> What I did was that I've extracted and deleted the rather huge OpenOffice >> source tree several times (sometimes with overwriting) and finally it >> ended up with these undeletable files and xfs errors. Is it considered >> normal for xfs to get messed up like that under such load? > > No, not normal. > > It found the text "0xAAAA0000,6,0x" on disk in an area where it expected > to find valid filesystem metadata. > > Corruption could come from anywhere - an xfs bug, some other bug, bad > memory, bad cables, neon death rays from space, writing directly to the > disk, who knows. Awfully hard to track down a one-off occurrence like > this, I'm afraid. > >> Thanks, >> Sab >> >> >> >> The xfs_repair output and xfs_info output is included below: >> >> # xfs_repair -v /dev/sda10 > > ... > >> Phase 6 - check inode connectivity... >> - resetting contents of realtime bitmap and summary inodes >> - traversing filesystem ... >> - agno = 0 >> - agno = 1 >> - agno = 2 >> - agno = 3 >> - agno = 4 >> - agno = 5 >> - agno = 6 >> - agno = 7 >> leaf block 8388608 for directory inode 4051737 bad header >> rebuilding directory inode 4051737 >> leaf block 8388608 for directory inode 4053318 bad header >> rebuilding directory inode 4053318 > > ... > > above is the problem, properly found & repaired. > > -Eric > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2009-09-02 21:01 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-09-02 13:22 Structure needs cleaning? (take #2) RUMI Szabolcs 2009-09-02 14:31 ` Eric Sandeen 2009-09-02 19:34 ` RUMI Szabolcs 2009-09-02 19:53 ` Eric Sandeen 2009-09-02 21:02 ` Justin Piszcz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox