corrupt file system -- "Structure needs cleaning"

corrupt file system -- "Structure needs cleaning"

[Hendrik Hoeth]

Hi,

I'm running linux-2.6.29.3 on a VIA Esther CPU. The harddisk is fully
encrypted using dm-crypt, and inside the encryption I have LVM with my
actual partitions. The file system is XFS, I have xfsprogs-2.9.4-1.

I was copying some large files when I got these errors (and yes, I own
that music CD ;-)):

-------------------8<---------------------
cp: cannot create regular file `maria/Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/03 Piano Concerto no. 1 in G minor op. 25 III. Presto.mp3': Structure needs cleaning
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/04 Variations serieuses op. 54.mp3': Input/output error
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/05 Rondo capriccioso op. 14.mp3': Input/output error
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/06 Piano Concerto no. 2 in D minor op. 40 I. Allegro appassionato.mp3': Input/output error
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/07 Piano Concerto no. 2 in D minor op. 40 II. Adagio.mp3': Input/output error
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/08 Piano Concerto no. 2 in D minor op. 40 III. Presto scherzando.mp3': Input/output error
cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/mendelssohn.png': Input/output error
[14:35] hoeth@jetway:~/Musik/DONE $ df -h .
df: `.'
df: no file systems processed
[14:36] hoeth@jetway:~/Musik/DONE $ ls
ls: cannot open directory .: Input/output error
[14:36] hoeth@jetway:~/Musik/DONE $ 
-------------------8<---------------------

So at this point I realised that the filesystem was shut down.
Here's what I see in the logfiles:

-------------------8<---------------------
Jun 26 14:35:44 jetway kernel: Filesystem "dm-5": XFS internal error xfs_btree_check_sblock at line 124 of file fs/xfs/xfs_btree.c.  Caller 0xc02201ac
Jun 26 14:35:44 jetway kernel: 
Jun 26 14:35:44 jetway kernel: Pid: 290, comm: pdflush Not tainted 2.6.29.3 #1
Jun 26 14:35:44 jetway kernel: Call Trace:
Jun 26 14:35:44 jetway kernel:  [<c023117e>] xfs_error_report+0x4e/0x50
Jun 26 14:35:44 jetway kernel:  [<c02201ac>] ? xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:44 jetway kernel:  [<c021fae6>] xfs_btree_check_sblock+0x56/0xe0
Jun 26 14:35:44 jetway kernel:  [<c02201ac>] ? xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:44 jetway kernel:  [<c02201ac>] xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:44 jetway kernel:  [<c022023f>] xfs_btree_read_buf_block+0x7f/0xa0
Jun 26 14:35:44 jetway kernel:  [<c02202d2>] xfs_btree_lookup_get_block+0x72/0xc0
Jun 26 14:35:44 jetway kernel:  [<c0221a2f>] xfs_btree_lookup+0x7f/0x3c0
Jun 26 14:35:44 jetway kernel:  [<c0253c17>] ? kmem_zone_zalloc+0x27/0x60
Jun 26 14:35:44 jetway kernel:  [<c020ad46>] xfs_alloc_lookup_ge+0x16/0x20
Jun 26 14:35:44 jetway kernel:  [<c020beda>] xfs_alloc_ag_vextent_near+0x4a/0x940
Jun 26 14:35:44 jetway kernel:  [<c024d920>] ? xfs_trans_read_buf+0x200/0x310
Jun 26 14:35:44 jetway kernel:  [<c020ced7>] xfs_alloc_ag_vextent+0xa7/0x100
Jun 26 14:35:44 jetway kernel:  [<c020d677>] xfs_alloc_vextent+0x227/0x410
Jun 26 14:35:44 jetway kernel:  [<c021c917>] xfs_bmap_btalloc+0x4f7/0x950
Jun 26 14:35:44 jetway kernel:  [<c021cd78>] xfs_bmap_alloc+0x8/0x10
Jun 26 14:35:44 jetway kernel:  [<c021d9f0>] xfs_bmapi+0xc70/0x1250
Jun 26 14:35:44 jetway kernel:  [<c0253b94>] ? kmem_zone_alloc+0x54/0xb0
Jun 26 14:35:44 jetway kernel:  [<c02413de>] ? xfs_log_reserve+0x7e/0xb0
Jun 26 14:35:44 jetway kernel:  [<c023bf2b>] xfs_iomap_write_allocate+0x24b/0x3c0
Jun 26 14:35:44 jetway kernel:  [<c023ce1e>] xfs_iomap+0x2ce/0x370
Jun 26 14:35:44 jetway kernel:  [<c0254803>] xfs_map_blocks+0x33/0x40
Jun 26 14:35:44 jetway kernel:  [<c02557eb>] xfs_page_state_convert+0x2db/0x6f0
Jun 26 14:35:44 jetway kernel:  [<c0255d18>] xfs_vm_writepage+0x58/0xe0
Jun 26 14:35:44 jetway kernel:  [<c014425b>] __writepage+0xb/0x30
Jun 26 14:35:44 jetway kernel:  [<c014448c>] write_cache_pages+0x1bc/0x360
Jun 26 14:35:44 jetway kernel:  [<c0144250>] ? __writepage+0x0/0x30
Jun 26 14:35:44 jetway kernel:  [<c0144653>] generic_writepages+0x23/0x30
Jun 26 14:35:44 jetway kernel:  [<c0254878>] xfs_vm_writepages+0x18/0x20
Jun 26 14:35:44 jetway kernel:  [<c014468e>] do_writepages+0x2e/0x50
Jun 26 14:35:44 jetway kernel:  [<c0176b70>] __writeback_single_inode+0x80/0x320
Jun 26 14:35:44 jetway kernel:  [<c01771ea>] generic_sync_sb_inodes+0x22a/0x2e0
Jun 26 14:35:44 jetway kernel:  [<c0253634>] ? xfs_bwrite+0x54/0xc0
Jun 26 14:35:44 jetway kernel:  [<c025ec84>] ? xfs_sync_fsdata+0x84/0xd0
Jun 26 14:35:44 jetway kernel:  [<c01772a8>] sync_sb_inodes+0x8/0x10
Jun 26 14:35:44 jetway kernel:  [<c0177432>] writeback_inodes+0x72/0x90
Jun 26 14:35:44 jetway kernel:  [<c0145222>] wb_kupdate+0x72/0xe0
Jun 26 14:35:44 jetway kernel:  [<c01455e0>] ? pdflush+0x0/0x180
Jun 26 14:35:44 jetway kernel:  [<c01456b2>] pdflush+0xd2/0x180
Jun 26 14:35:44 jetway kernel:  [<c01451b0>] ? wb_kupdate+0x0/0xe0
Jun 26 14:35:44 jetway kernel:  [<c012d422>] kthread+0x42/0x70
Jun 26 14:35:44 jetway kernel:  [<c012d3e0>] ? kthread+0x0/0x70
Jun 26 14:35:44 jetway kernel:  [<c010379f>] kernel_thread_helper+0x7/0x18
Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": XFS internal error xfs_btree_check_sblock at line 124 of file fs/xfs/xfs_btree.c.  Caller 0xc02201ac
Jun 26 14:35:45 jetway kernel: 
Jun 26 14:35:45 jetway kernel: Pid: 2618, comm: cp Not tainted 2.6.29.3 #1
Jun 26 14:35:45 jetway kernel: Call Trace:
Jun 26 14:35:45 jetway kernel:  [<c023117e>] xfs_error_report+0x4e/0x50
Jun 26 14:35:45 jetway kernel:  [<c02201ac>] ? xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:45 jetway kernel:  [<c021fae6>] xfs_btree_check_sblock+0x56/0xe0
Jun 26 14:35:45 jetway kernel:  [<c02201ac>] ? xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:45 jetway kernel:  [<c02201ac>] xfs_btree_check_block+0x2c/0x40
Jun 26 14:35:45 jetway kernel:  [<c022023f>] xfs_btree_read_buf_block+0x7f/0xa0
Jun 26 14:35:45 jetway kernel:  [<c02202d2>] xfs_btree_lookup_get_block+0x72/0xc0
Jun 26 14:35:45 jetway kernel:  [<c0221a2f>] xfs_btree_lookup+0x7f/0x3c0
Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ? kmem_zone_zalloc+0x27/0x60
Jun 26 14:35:45 jetway kernel:  [<c020ad46>] xfs_alloc_lookup_ge+0x16/0x20
Jun 26 14:35:45 jetway kernel:  [<c020beda>] xfs_alloc_ag_vextent_near+0x4a/0x940
Jun 26 14:35:45 jetway kernel:  [<c020ced7>] xfs_alloc_ag_vextent+0xa7/0x100
Jun 26 14:35:45 jetway kernel:  [<c020d677>] xfs_alloc_vextent+0x227/0x410
Jun 26 14:35:45 jetway kernel:  [<c021c917>] xfs_bmap_btalloc+0x4f7/0x950
Jun 26 14:35:45 jetway kernel:  [<c023b99b>] ? xfs_iomap_eof_want_preallocate+0x12b/0x1c0
Jun 26 14:35:45 jetway kernel:  [<c021cd78>] xfs_bmap_alloc+0x8/0x10
Jun 26 14:35:45 jetway kernel:  [<c021d9f0>] xfs_bmapi+0xc70/0x1250
Jun 26 14:35:45 jetway kernel:  [<c01405cd>] ? find_or_create_page+0x2d/0x90
Jun 26 14:35:45 jetway kernel:  [<c0228125>] xfs_dir2_grow_inode+0x115/0x3d0
Jun 26 14:35:45 jetway kernel:  [<c0253b94>] ? kmem_zone_alloc+0x54/0xb0
Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ? kmem_zone_zalloc+0x27/0x60
Jun 26 14:35:45 jetway kernel:  [<c0253c82>] ? kmem_free+0x32/0x50
Jun 26 14:35:45 jetway kernel:  [<c0238b95>] ? xfs_idata_realloc+0x35/0x150
Jun 26 14:35:45 jetway kernel:  [<c0229177>] xfs_dir2_sf_to_block+0x97/0x580
Jun 26 14:35:45 jetway kernel:  [<c02563d1>] ? xfs_buf_rele+0x51/0x70
Jun 26 14:35:45 jetway kernel:  [<c024dad9>] ? xfs_trans_brelse+0xa9/0xe0
Jun 26 14:35:45 jetway kernel:  [<c0253b94>] ? kmem_zone_alloc+0x54/0xb0
Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ? kmem_zone_zalloc+0x27/0x60
Jun 26 14:35:45 jetway kernel:  [<c023000d>] xfs_dir2_sf_addname+0xad/0x5b0
Jun 26 14:35:45 jetway kernel:  [<c016eefc>] ? unlock_new_inode+0x2c/0x50
Jun 26 14:35:45 jetway kernel:  [<c0170171>] ? inode_add_to_lists+0x11/0x70
Jun 26 14:35:45 jetway kernel:  [<c025a5bf>] ? xfs_setup_inode+0x14f/0x200
Jun 26 14:35:45 jetway kernel:  [<c0228908>] xfs_dir_createname+0x108/0x120
Jun 26 14:35:45 jetway kernel:  [<c0250b7a>] xfs_create+0x31a/0x430
Jun 26 14:35:45 jetway kernel:  [<c025b248>] xfs_vn_mknod+0x118/0x210
Jun 26 14:35:45 jetway kernel:  [<c025b372>] xfs_vn_create+0x12/0x20
Jun 26 14:35:45 jetway kernel:  [<c0167070>] vfs_create+0x80/0xc0
Jun 26 14:35:45 jetway kernel:  [<c025b360>] ? xfs_vn_create+0x0/0x20
Jun 26 14:35:45 jetway kernel:  [<c01699de>] do_filp_open+0x60e/0x6e0
Jun 26 14:35:45 jetway kernel:  [<c015debb>] do_sys_open+0x4b/0xe0
Jun 26 14:35:45 jetway kernel:  [<c015dfb9>] sys_open+0x29/0x40
Jun 26 14:35:45 jetway kernel:  [<c0103145>] sysenter_do_call+0x12/0x25
Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": XFS internal error xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c.  Caller 0xc02509c3
Jun 26 14:35:45 jetway kernel: 
Jun 26 14:35:45 jetway kernel: Pid: 2618, comm: cp Not tainted 2.6.29.3 #1
Jun 26 14:35:45 jetway kernel: Call Trace:
Jun 26 14:35:45 jetway kernel:  [<c023117e>] xfs_error_report+0x4e/0x50
Jun 26 14:35:45 jetway kernel:  [<c02509c3>] ? xfs_create+0x163/0x430
Jun 26 14:35:45 jetway kernel:  [<c024c101>] xfs_trans_cancel+0xd1/0xf0
Jun 26 14:35:45 jetway kernel:  [<c02509c3>] ? xfs_create+0x163/0x430
Jun 26 14:35:45 jetway kernel:  [<c02509c3>] xfs_create+0x163/0x430
Jun 26 14:35:45 jetway kernel:  [<c025b248>] xfs_vn_mknod+0x118/0x210
Jun 26 14:35:45 jetway kernel:  [<c025b372>] xfs_vn_create+0x12/0x20
Jun 26 14:35:45 jetway kernel:  [<c0167070>] vfs_create+0x80/0xc0
Jun 26 14:35:45 jetway kernel:  [<c025b360>] ? xfs_vn_create+0x0/0x20
Jun 26 14:35:45 jetway kernel:  [<c01699de>] do_filp_open+0x60e/0x6e0
Jun 26 14:35:45 jetway kernel:  [<c015debb>] do_sys_open+0x4b/0xe0
Jun 26 14:35:45 jetway kernel:  [<c015dfb9>] sys_open+0x29/0x40
Jun 26 14:35:45 jetway kernel:  [<c0103145>] sysenter_do_call+0x12/0x25
Jun 26 14:35:45 jetway kernel: xfs_force_shutdown(dm-5,0x8) called from line 1165 of file fs/xfs/xfs_trans.c.  Return address = 0xc024c119
Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": Corruption of in-memory data detected.  Shutting down filesystem: dm-5
Jun 26 14:35:45 jetway kernel: Please umount the filesystem, and rectify the problem(s)
Jun 26 14:35:48 jetway kernel: Filesystem "dm-5": xfs_log_force: error 5 returned.
Jun 26 14:36:18 jetway kernel: Filesystem "dm-5": xfs_log_force: error 5 returned.
Jun 26 14:37:18 jetway last message repeated 2 times
Jun 26 14:38:18 jetway last message repeated 2 times
Jun 26 14:39:18 jetway last message repeated 2 times
Jun 26 14:40:18 jetway last message repeated 2 times
Jun 26 14:41:18 jetway last message repeated 2 times
Jun 26 14:41:55 jetway last message repeated 6 times
Jun 26 14:43:56 jetway kernel: Filesystem "dm-5": Disabling barriers, trial barrier write failed
Jun 26 14:43:56 jetway kernel: XFS mounting filesystem dm-5
Jun 26 14:43:57 jetway kernel: Starting XFS recovery on filesystem: dm-5 (logdev: internal)
Jun 26 14:43:57 jetway kernel: Ending XFS recovery on filesystem: dm-5 (logdev: internal)
Jun 26 14:44:18 jetway kernel: Filesystem "dm-5": Disabling barriers, trial barrier write failed
Jun 26 14:44:18 jetway kernel: XFS mounting filesystem dm-5
Jun 26 14:44:18 jetway kernel: Ending clean XFS mount for filesystem: dm-5
-------------------8<---------------------

This is how I recovered (well, most of the data I had copied is
corrupt at the target location):

-------------------8<---------------------
[14:40] root@jetway:/var/log # umount /home 
[14:43] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_check.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
[14:43] root@jetway:/var/log # mount /home/
[14:43] root@jetway:/var/log # umount /home/
[14:44] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
[14:44] root@jetway:/var/log # mount /home/
-------------------8<---------------------

Please let me know if you want to have any further information.
I'm not on the mailing list, so Cc me directly.

Cheers,

    Hendrik

-- 
"You have to take the most direct road to go instead of your 
 meeting, you have to, this one ended, leave at once the CERN
 domain."         (imprint on the CERN visitor ID cards)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Eric Sandeen]

Hendrik Hoeth wrote:
> Hi,
> 
> I'm running linux-2.6.29.3 on a VIA Esther CPU. The harddisk is fully
> encrypted using dm-crypt, and inside the encryption I have LVM with my
> actual partitions. The file system is XFS, I have xfsprogs-2.9.4-1.
> 
> I was copying some large files when I got these errors (and yes, I own
> that music CD ;-)):
> 
> -------------------8<---------------------
> cp: cannot create regular file

...

> [14:36] hoeth@jetway:~/Musik/DONE $ ls
> ls: cannot open directory .: Input/output error
> [14:36] hoeth@jetway:~/Musik/DONE $ 
> -------------------8<---------------------
> 
> So at this point I realised that the filesystem was shut down.

Yep

> Here's what I see in the logfiles:
> 
> -------------------8<---------------------
> Jun 26 14:35:44 jetway kernel: Filesystem "dm-5": XFS internal error xfs_btree_check_sblock at line 124 of file fs/xfs/xfs_btree.c.  Caller 0xc02201ac

This is an internal consistency check failing

> -------------------8<---------------------
> 
> This is how I recovered (well, most of the data I had copied is
> corrupt at the target location):
> 
> -------------------8<---------------------
> [14:40] root@jetway:/var/log # umount /home 
> [14:43] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
> ERROR: The filesystem has valuable metadata changes in a log which needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it before
> re-running xfs_check.  If you are unable to mount the filesystem, then use
> the xfs_repair -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a mount
> of the filesystem before doing this.
> [14:43] root@jetway:/var/log # mount /home/
> [14:43] root@jetway:/var/log # umount /home/
> [14:44] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home

xfs_check doesn't actually fix anything; I'd run xfs_repair.  Use -n
first if you want to see what it would do.

If it doesn't find anything, then I guess you had some in-memory corruption.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Felix Blyakher]

On Jun 26, 2009, at 2:58 PM, Hendrik Hoeth wrote:

>
> This is how I recovered (well, most of the data I had copied is
> corrupt at the target location):

Some files would not make it to disk when filesystem shut down.
I'd expect that such files are short. Do you have other indication
that the files are corrupt?

> -------------------8<---------------------
> [14:40] root@jetway:/var/log # umount /home
> [14:43] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
> ERROR: The filesystem has valuable metadata changes in a log which  
> needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it  
> before
> re-running xfs_check.  If you are unable to mount the filesystem,  
> then use
> the xfs_repair -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt  
> a mount
> of the filesystem before doing this.
> [14:43] root@jetway:/var/log # mount /home/
> [14:43] root@jetway:/var/log # umount /home/
> [14:44] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
> [14:44] root@jetway:/var/log # mount /home/
> -------------------8<---------------------

Seems like no corruption on disk.
The system detected in memory corruption and shut the filesystem
down to avoid writing bad (meta)data to disk. I don't have answer
for that problem, however you can continue to use the filesystem,
and may never see this problem again. If it happens again, it
could be useful to run debug XFS, which may point to the problem.

Thanks,
Felix

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Hendrik Hoeth]

Thus spake Felix Blyakher (felixb@sgi.com):

> On Jun 26, 2009, at 2:58 PM, Hendrik Hoeth wrote:
>>
>> This is how I recovered (well, most of the data I had copied is
>> corrupt at the target location):
>
> Some files would not make it to disk when filesystem shut down. I'd
> expect that such files are short. Do you have other indication that
> the files are corrupt?

Of course the last file was short, but even several of the files that
had been copied just before are corrupt (it was a "cp -r" of a few
directories, total ~ 700 MByte, the error appeared when copying the last
directory). Those files have identical lengths as the source files, but
different md5sum. Since it's music, I can hear the corrupt positions;
they are at random positions somewhere in the files.

> Seems like no corruption on disk.
> The system detected in memory corruption and shut the filesystem
> down to avoid writing bad (meta)data to disk. I don't have answer
> for that problem, however you can continue to use the filesystem,
> and may never see this problem again.

Hmmm, at least no corruption as in "inconsistent file system", right?
Still doesn't make me feel well ... I guess I should run a memtest.

> If it happens again, it could be useful to run debug XFS, which may
> point to the problem.

It's the first time that I had problems on this system, and it has been
running continuously for 1 1/2 years now. So until today I thought it
was stable. If the memtest doesn't show anything, I'll keep an eye on
this.

Thanks for the help,

     Hendrik

-- 
"You have to take the most direct road to go instead of your 
 meeting, you have to, this one ended, leave at once the CERN
 domain."         (imprint on the CERN visitor ID cards)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Hendrik Hoeth]

Thus spake Eric Sandeen (sandeen@sandeen.net):

> > [14:44] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
> 
> xfs_check doesn't actually fix anything; I'd run xfs_repair.  Use -n
> first if you want to see what it would do.

I thought that if xfs_check doesn't show any errors, the filesystem is
fine and doesn't need an xfs_repair. Is that wrong?

Anyway xfs_repair looks healthy to me:

-------------------8<---------------------
[23:23] root@jetway:~ # xfs_repair /dev/mapper/hda_crypt_vg-home
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - agno = 4
        - agno = 5
        - agno = 6
        - agno = 7
        - agno = 8
        - agno = 9
        - agno = 10
        - agno = 11
        - agno = 12
        - agno = 13
        - agno = 14
        - agno = 15
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
[23:23] root@jetway:~ # 
-------------------8<---------------------

> If it doesn't find anything, then I guess you had some in-memory corruption.

I'll run memtest.

Thanks,

    Hendrik

-- 
"You have to take the most direct road to go instead of your 
 meeting, you have to, this one ended, leave at once the CERN
 domain."         (imprint on the CERN visitor ID cards)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Eric Sandeen]

Hendrik Hoeth wrote:
> Thus spake Eric Sandeen (sandeen@sandeen.net):
> 
>>> [14:44] root@jetway:/var/log # xfs_check /dev/mapper/hda_crypt_vg-home
>> xfs_check doesn't actually fix anything; I'd run xfs_repair.  Use -n
>> first if you want to see what it would do.
> 
> I thought that if xfs_check doesn't show any errors, the filesystem is
> fine and doesn't need an xfs_repair. Is that wrong?

Well, yes, I suppose that's true.

I almost never use xfs_check mostly because it's a memory hog, still,
and xfs_repair checks almost everything that xfs_check does (and what
isn't checked is rebuilt anyway)

> Anyway xfs_repair looks healthy to me:

good news, I guess :)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Hendrik Hoeth]

Thus spake Felix Blyakher (felixb@sgi.com):

> Seems like no corruption on disk.
> The system detected in memory corruption and shut the filesystem
> down to avoid writing bad (meta)data to disk. I don't have answer
> for that problem, however you can continue to use the filesystem,
> and may never see this problem again. If it happens again, it
> could be useful to run debug XFS, which may point to the problem.

I ran memtest over night, without any errors. So if this happens again
and I need to run xfs_db, what do I need to do to get useful information
out? Haven't done that before ...

    Hendrik

-- 
"You have to take the most direct road to go instead of your 
 meeting, you have to, this one ended, leave at once the CERN
 domain."         (imprint on the CERN visitor ID cards)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Felix Blyakher]

On Jun 27, 2009, at 2:08 AM, Hendrik Hoeth wrote:

> Thus spake Felix Blyakher (felixb@sgi.com):
>
>> Seems like no corruption on disk.
>> The system detected in memory corruption and shut the filesystem
>> down to avoid writing bad (meta)data to disk. I don't have answer
>> for that problem, however you can continue to use the filesystem,
>> and may never see this problem again. If it happens again, it
>> could be useful to run debug XFS, which may point to the problem.
>
> I ran memtest over night, without any errors. So if this happens again
> and I need to run xfs_db,

No, I didn't mean xfs_db. That examines the data structures on disk,
but apparently they're clean anyway.

> what do I need to do to get useful information
> out? Haven't done that before ...

What I meant is to rebuild the XFS with the CONFIG_XFS_DEBUG set.
With that option XFS has much more checks in place and hopefully
will assert earlier, giving us more clues.
It would be great help to XFS developers, but I'm not sure if it's
too much to ask for.

Thanks,
Felix

>
>
>    Hendrik
>
> -- 
> "You have to take the most direct road to go instead of your
> meeting, you have to, this one ended, leave at once the CERN
> domain."         (imprint on the CERN visitor ID cards)
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: corrupt file system -- "Structure needs cleaning"

[Lachlan McIlroy]

Hi Hendrik,

This looks similar to another issue I've seen before where the "Structure
needs cleaning" error was reported when trying to read an inode cluster.
We observed that an I/O was issued to read in an inode cluster but the
pages we got back did not contain the expected magic numbers for inodes
- the pages were mostly full of zeroes.  What was on disk was correct so
it looked like the I/O completely prematurely or the data was not read
into the correct location.  We never got to the root cause of the problem
but we did have a workaround that detected when the magic numbers were not
correct, invalidated the buffer and reissued the I/O.  The re-issued I/O
always read in the correct data.

Could you set this tunable and see if it produces more information the next
time you see this problem?

$ echo 11 > /proc/sys/fs/xfs/error_level

Part of our workaround was to use this patch:

http://oss.sgi.com/archives/xfs/2009-02/msg00177.html

Would you mind trying it?

Lachlan

----- "Hendrik Hoeth" <hendrik.hoeth@cern.ch> wrote:

> Hi,
> 
> I'm running linux-2.6.29.3 on a VIA Esther CPU. The harddisk is fully
> encrypted using dm-crypt, and inside the encryption I have LVM with
> my
> actual partitions. The file system is XFS, I have xfsprogs-2.9.4-1.
> 
> I was copying some large files when I got these errors (and yes, I
> own
> that music CD ;-)):
> 
> -------------------8<---------------------
> cp: cannot create regular file `maria/Mendelssohn, Felix -
> Klavierkonzerte (Jean-Yves Thibaudet, Gewandhausorchester)/03 Piano
> Concerto no. 1 in G minor op. 25 III. Presto.mp3': Structure needs
> cleaning
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/04 Variations serieuses op. 54.mp3':
> Input/output error
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/05 Rondo capriccioso op. 14.mp3':
> Input/output error
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/06 Piano Concerto no. 2 in D minor op.
> 40 I. Allegro appassionato.mp3': Input/output error
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/07 Piano Concerto no. 2 in D minor op.
> 40 II. Adagio.mp3': Input/output error
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/08 Piano Concerto no. 2 in D minor op.
> 40 III. Presto scherzando.mp3': Input/output error
> cp: cannot stat `Mendelssohn, Felix - Klavierkonzerte (Jean-Yves
> Thibaudet, Gewandhausorchester)/mendelssohn.png': Input/output error
> [14:35] hoeth@jetway:~/Musik/DONE $ df -h .
> df: `.'
> df: no file systems processed
> [14:36] hoeth@jetway:~/Musik/DONE $ ls
> ls: cannot open directory .: Input/output error
> [14:36] hoeth@jetway:~/Musik/DONE $ 
> -------------------8<---------------------
> 
> So at this point I realised that the filesystem was shut down.
> Here's what I see in the logfiles:
> 
> -------------------8<---------------------
> Jun 26 14:35:44 jetway kernel: Filesystem "dm-5": XFS internal error
> xfs_btree_check_sblock at line 124 of file fs/xfs/xfs_btree.c.  Caller
> 0xc02201ac
> Jun 26 14:35:44 jetway kernel: 
> Jun 26 14:35:44 jetway kernel: Pid: 290, comm: pdflush Not tainted
> 2.6.29.3 #1
> Jun 26 14:35:44 jetway kernel: Call Trace:
> Jun 26 14:35:44 jetway kernel:  [<c023117e>]
> xfs_error_report+0x4e/0x50
> Jun 26 14:35:44 jetway kernel:  [<c02201ac>] ?
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:44 jetway kernel:  [<c021fae6>]
> xfs_btree_check_sblock+0x56/0xe0
> Jun 26 14:35:44 jetway kernel:  [<c02201ac>] ?
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:44 jetway kernel:  [<c02201ac>]
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:44 jetway kernel:  [<c022023f>]
> xfs_btree_read_buf_block+0x7f/0xa0
> Jun 26 14:35:44 jetway kernel:  [<c02202d2>]
> xfs_btree_lookup_get_block+0x72/0xc0
> Jun 26 14:35:44 jetway kernel:  [<c0221a2f>]
> xfs_btree_lookup+0x7f/0x3c0
> Jun 26 14:35:44 jetway kernel:  [<c0253c17>] ?
> kmem_zone_zalloc+0x27/0x60
> Jun 26 14:35:44 jetway kernel:  [<c020ad46>]
> xfs_alloc_lookup_ge+0x16/0x20
> Jun 26 14:35:44 jetway kernel:  [<c020beda>]
> xfs_alloc_ag_vextent_near+0x4a/0x940
> Jun 26 14:35:44 jetway kernel:  [<c024d920>] ?
> xfs_trans_read_buf+0x200/0x310
> Jun 26 14:35:44 jetway kernel:  [<c020ced7>]
> xfs_alloc_ag_vextent+0xa7/0x100
> Jun 26 14:35:44 jetway kernel:  [<c020d677>]
> xfs_alloc_vextent+0x227/0x410
> Jun 26 14:35:44 jetway kernel:  [<c021c917>]
> xfs_bmap_btalloc+0x4f7/0x950
> Jun 26 14:35:44 jetway kernel:  [<c021cd78>] xfs_bmap_alloc+0x8/0x10
> Jun 26 14:35:44 jetway kernel:  [<c021d9f0>] xfs_bmapi+0xc70/0x1250
> Jun 26 14:35:44 jetway kernel:  [<c0253b94>] ?
> kmem_zone_alloc+0x54/0xb0
> Jun 26 14:35:44 jetway kernel:  [<c02413de>] ?
> xfs_log_reserve+0x7e/0xb0
> Jun 26 14:35:44 jetway kernel:  [<c023bf2b>]
> xfs_iomap_write_allocate+0x24b/0x3c0
> Jun 26 14:35:44 jetway kernel:  [<c023ce1e>] xfs_iomap+0x2ce/0x370
> Jun 26 14:35:44 jetway kernel:  [<c0254803>] xfs_map_blocks+0x33/0x40
> Jun 26 14:35:44 jetway kernel:  [<c02557eb>]
> xfs_page_state_convert+0x2db/0x6f0
> Jun 26 14:35:44 jetway kernel:  [<c0255d18>]
> xfs_vm_writepage+0x58/0xe0
> Jun 26 14:35:44 jetway kernel:  [<c014425b>] __writepage+0xb/0x30
> Jun 26 14:35:44 jetway kernel:  [<c014448c>]
> write_cache_pages+0x1bc/0x360
> Jun 26 14:35:44 jetway kernel:  [<c0144250>] ? __writepage+0x0/0x30
> Jun 26 14:35:44 jetway kernel:  [<c0144653>]
> generic_writepages+0x23/0x30
> Jun 26 14:35:44 jetway kernel:  [<c0254878>]
> xfs_vm_writepages+0x18/0x20
> Jun 26 14:35:44 jetway kernel:  [<c014468e>] do_writepages+0x2e/0x50
> Jun 26 14:35:44 jetway kernel:  [<c0176b70>]
> __writeback_single_inode+0x80/0x320
> Jun 26 14:35:44 jetway kernel:  [<c01771ea>]
> generic_sync_sb_inodes+0x22a/0x2e0
> Jun 26 14:35:44 jetway kernel:  [<c0253634>] ? xfs_bwrite+0x54/0xc0
> Jun 26 14:35:44 jetway kernel:  [<c025ec84>] ?
> xfs_sync_fsdata+0x84/0xd0
> Jun 26 14:35:44 jetway kernel:  [<c01772a8>] sync_sb_inodes+0x8/0x10
> Jun 26 14:35:44 jetway kernel:  [<c0177432>]
> writeback_inodes+0x72/0x90
> Jun 26 14:35:44 jetway kernel:  [<c0145222>] wb_kupdate+0x72/0xe0
> Jun 26 14:35:44 jetway kernel:  [<c01455e0>] ? pdflush+0x0/0x180
> Jun 26 14:35:44 jetway kernel:  [<c01456b2>] pdflush+0xd2/0x180
> Jun 26 14:35:44 jetway kernel:  [<c01451b0>] ? wb_kupdate+0x0/0xe0
> Jun 26 14:35:44 jetway kernel:  [<c012d422>] kthread+0x42/0x70
> Jun 26 14:35:44 jetway kernel:  [<c012d3e0>] ? kthread+0x0/0x70
> Jun 26 14:35:44 jetway kernel:  [<c010379f>]
> kernel_thread_helper+0x7/0x18
> Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": XFS internal error
> xfs_btree_check_sblock at line 124 of file fs/xfs/xfs_btree.c.  Caller
> 0xc02201ac
> Jun 26 14:35:45 jetway kernel: 
> Jun 26 14:35:45 jetway kernel: Pid: 2618, comm: cp Not tainted
> 2.6.29.3 #1
> Jun 26 14:35:45 jetway kernel: Call Trace:
> Jun 26 14:35:45 jetway kernel:  [<c023117e>]
> xfs_error_report+0x4e/0x50
> Jun 26 14:35:45 jetway kernel:  [<c02201ac>] ?
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:45 jetway kernel:  [<c021fae6>]
> xfs_btree_check_sblock+0x56/0xe0
> Jun 26 14:35:45 jetway kernel:  [<c02201ac>] ?
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:45 jetway kernel:  [<c02201ac>]
> xfs_btree_check_block+0x2c/0x40
> Jun 26 14:35:45 jetway kernel:  [<c022023f>]
> xfs_btree_read_buf_block+0x7f/0xa0
> Jun 26 14:35:45 jetway kernel:  [<c02202d2>]
> xfs_btree_lookup_get_block+0x72/0xc0
> Jun 26 14:35:45 jetway kernel:  [<c0221a2f>]
> xfs_btree_lookup+0x7f/0x3c0
> Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ?
> kmem_zone_zalloc+0x27/0x60
> Jun 26 14:35:45 jetway kernel:  [<c020ad46>]
> xfs_alloc_lookup_ge+0x16/0x20
> Jun 26 14:35:45 jetway kernel:  [<c020beda>]
> xfs_alloc_ag_vextent_near+0x4a/0x940
> Jun 26 14:35:45 jetway kernel:  [<c020ced7>]
> xfs_alloc_ag_vextent+0xa7/0x100
> Jun 26 14:35:45 jetway kernel:  [<c020d677>]
> xfs_alloc_vextent+0x227/0x410
> Jun 26 14:35:45 jetway kernel:  [<c021c917>]
> xfs_bmap_btalloc+0x4f7/0x950
> Jun 26 14:35:45 jetway kernel:  [<c023b99b>] ?
> xfs_iomap_eof_want_preallocate+0x12b/0x1c0
> Jun 26 14:35:45 jetway kernel:  [<c021cd78>] xfs_bmap_alloc+0x8/0x10
> Jun 26 14:35:45 jetway kernel:  [<c021d9f0>] xfs_bmapi+0xc70/0x1250
> Jun 26 14:35:45 jetway kernel:  [<c01405cd>] ?
> find_or_create_page+0x2d/0x90
> Jun 26 14:35:45 jetway kernel:  [<c0228125>]
> xfs_dir2_grow_inode+0x115/0x3d0
> Jun 26 14:35:45 jetway kernel:  [<c0253b94>] ?
> kmem_zone_alloc+0x54/0xb0
> Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ?
> kmem_zone_zalloc+0x27/0x60
> Jun 26 14:35:45 jetway kernel:  [<c0253c82>] ? kmem_free+0x32/0x50
> Jun 26 14:35:45 jetway kernel:  [<c0238b95>] ?
> xfs_idata_realloc+0x35/0x150
> Jun 26 14:35:45 jetway kernel:  [<c0229177>]
> xfs_dir2_sf_to_block+0x97/0x580
> Jun 26 14:35:45 jetway kernel:  [<c02563d1>] ? xfs_buf_rele+0x51/0x70
> Jun 26 14:35:45 jetway kernel:  [<c024dad9>] ?
> xfs_trans_brelse+0xa9/0xe0
> Jun 26 14:35:45 jetway kernel:  [<c0253b94>] ?
> kmem_zone_alloc+0x54/0xb0
> Jun 26 14:35:45 jetway kernel:  [<c0253c17>] ?
> kmem_zone_zalloc+0x27/0x60
> Jun 26 14:35:45 jetway kernel:  [<c023000d>]
> xfs_dir2_sf_addname+0xad/0x5b0
> Jun 26 14:35:45 jetway kernel:  [<c016eefc>] ?
> unlock_new_inode+0x2c/0x50
> Jun 26 14:35:45 jetway kernel:  [<c0170171>] ?
> inode_add_to_lists+0x11/0x70
> Jun 26 14:35:45 jetway kernel:  [<c025a5bf>] ?
> xfs_setup_inode+0x14f/0x200
> Jun 26 14:35:45 jetway kernel:  [<c0228908>]
> xfs_dir_createname+0x108/0x120
> Jun 26 14:35:45 jetway kernel:  [<c0250b7a>] xfs_create+0x31a/0x430
> Jun 26 14:35:45 jetway kernel:  [<c025b248>] xfs_vn_mknod+0x118/0x210
> Jun 26 14:35:45 jetway kernel:  [<c025b372>] xfs_vn_create+0x12/0x20
> Jun 26 14:35:45 jetway kernel:  [<c0167070>] vfs_create+0x80/0xc0
> Jun 26 14:35:45 jetway kernel:  [<c025b360>] ? xfs_vn_create+0x0/0x20
> Jun 26 14:35:45 jetway kernel:  [<c01699de>] do_filp_open+0x60e/0x6e0
> Jun 26 14:35:45 jetway kernel:  [<c015debb>] do_sys_open+0x4b/0xe0
> Jun 26 14:35:45 jetway kernel:  [<c015dfb9>] sys_open+0x29/0x40
> Jun 26 14:35:45 jetway kernel:  [<c0103145>]
> sysenter_do_call+0x12/0x25
> Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": XFS internal error
> xfs_trans_cancel at line 1164 of file fs/xfs/xfs_trans.c.  Caller
> 0xc02509c3
> Jun 26 14:35:45 jetway kernel: 
> Jun 26 14:35:45 jetway kernel: Pid: 2618, comm: cp Not tainted
> 2.6.29.3 #1
> Jun 26 14:35:45 jetway kernel: Call Trace:
> Jun 26 14:35:45 jetway kernel:  [<c023117e>]
> xfs_error_report+0x4e/0x50
> Jun 26 14:35:45 jetway kernel:  [<c02509c3>] ? xfs_create+0x163/0x430
> Jun 26 14:35:45 jetway kernel:  [<c024c101>]
> xfs_trans_cancel+0xd1/0xf0
> Jun 26 14:35:45 jetway kernel:  [<c02509c3>] ? xfs_create+0x163/0x430
> Jun 26 14:35:45 jetway kernel:  [<c02509c3>] xfs_create+0x163/0x430
> Jun 26 14:35:45 jetway kernel:  [<c025b248>] xfs_vn_mknod+0x118/0x210
> Jun 26 14:35:45 jetway kernel:  [<c025b372>] xfs_vn_create+0x12/0x20
> Jun 26 14:35:45 jetway kernel:  [<c0167070>] vfs_create+0x80/0xc0
> Jun 26 14:35:45 jetway kernel:  [<c025b360>] ? xfs_vn_create+0x0/0x20
> Jun 26 14:35:45 jetway kernel:  [<c01699de>] do_filp_open+0x60e/0x6e0
> Jun 26 14:35:45 jetway kernel:  [<c015debb>] do_sys_open+0x4b/0xe0
> Jun 26 14:35:45 jetway kernel:  [<c015dfb9>] sys_open+0x29/0x40
> Jun 26 14:35:45 jetway kernel:  [<c0103145>]
> sysenter_do_call+0x12/0x25
> Jun 26 14:35:45 jetway kernel: xfs_force_shutdown(dm-5,0x8) called
> from line 1165 of file fs/xfs/xfs_trans.c.  Return address =
> 0xc024c119
> Jun 26 14:35:45 jetway kernel: Filesystem "dm-5": Corruption of
> in-memory data detected.  Shutting down filesystem: dm-5
> Jun 26 14:35:45 jetway kernel: Please umount the filesystem, and
> rectify the problem(s)
> Jun 26 14:35:48 jetway kernel: Filesystem "dm-5": xfs_log_force: error
> 5 returned.
> Jun 26 14:36:18 jetway kernel: Filesystem "dm-5": xfs_log_force: error
> 5 returned.
> Jun 26 14:37:18 jetway last message repeated 2 times
> Jun 26 14:38:18 jetway last message repeated 2 times
> Jun 26 14:39:18 jetway last message repeated 2 times
> Jun 26 14:40:18 jetway last message repeated 2 times
> Jun 26 14:41:18 jetway last message repeated 2 times
> Jun 26 14:41:55 jetway last message repeated 6 times
> Jun 26 14:43:56 jetway kernel: Filesystem "dm-5": Disabling barriers,
> trial barrier write failed
> Jun 26 14:43:56 jetway kernel: XFS mounting filesystem dm-5
> Jun 26 14:43:57 jetway kernel: Starting XFS recovery on filesystem:
> dm-5 (logdev: internal)
> Jun 26 14:43:57 jetway kernel: Ending XFS recovery on filesystem: dm-5
> (logdev: internal)
> Jun 26 14:44:18 jetway kernel: Filesystem "dm-5": Disabling barriers,
> trial barrier write failed
> Jun 26 14:44:18 jetway kernel: XFS mounting filesystem dm-5
> Jun 26 14:44:18 jetway kernel: Ending clean XFS mount for filesystem:
> dm-5
> -------------------8<---------------------
> 
> This is how I recovered (well, most of the data I had copied is
> corrupt at the target location):
> 
> -------------------8<---------------------
> [14:40] root@jetway:/var/log # umount /home 
> [14:43] root@jetway:/var/log # xfs_check
> /dev/mapper/hda_crypt_vg-home
> ERROR: The filesystem has valuable metadata changes in a log which
> needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it
> before
> re-running xfs_check.  If you are unable to mount the filesystem, then
> use
> the xfs_repair -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a
> mount
> of the filesystem before doing this.
> [14:43] root@jetway:/var/log # mount /home/
> [14:43] root@jetway:/var/log # umount /home/
> [14:44] root@jetway:/var/log # xfs_check
> /dev/mapper/hda_crypt_vg-home
> [14:44] root@jetway:/var/log # mount /home/
> -------------------8<---------------------
> 
> Please let me know if you want to have any further information.
> I'm not on the mailing list, so Cc me directly.
> 
> Cheers,
> 
>     Hendrik
> 
> -- 
> "You have to take the most direct road to go instead of your 
>  meeting, you have to, this one ended, leave at once the CERN
>  domain."         (imprint on the CERN visitor ID cards)
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs