large filesystem corruptions

large filesystem corruptions

[Kapetanakis Giannis]

Hi,

The last few days I'm trying to setup a large filesystem
14TB and is always getting corrupted...

This is my latest non-working setup. Sorry for long message
but I want to make clear my  actual setup

2.6.18-164.11.1.el5PAE (x86)
4GB RAM

CONFIG_EFI_VARS=y
CONFIG_EFI=y

/dev/sdb - 6 x 1.5TB SATA drives in hardware RAID 5 (256 chunk size)
/dev/sdc - 6 x 1.5TB SATA drives in hardware RAID 5 (256 chunk size)

-- Both hardware raids are GPT labeled

Model: Adaptec ARRAY01 (scsi)
Disk /dev/sdb: 7489GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
  1      17.4kB  7489GB  7489GB               ARRAY01

Model: Adaptec ARRAY02 (scsi)
Disk /dev/sdc: 7489GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
  1      17.4kB  7489GB  7489GB               ARRAY02

-- /dev/md_d0 software raid0 on /dev/sdb1 and /dev/sdc1
-- (doing on /dev/sdb and /dev/sdc also corrupted)
-- this raid0 is also GPT labeled (256 chunk size)

mdadm --create /dev/md_d0 -a p1 -c 256 -l 0 -n 2 /dev/sdb1 /dev/sdc1

   md_d0 : active raid0 sdb1[0] sdc1[1]
       14627614208 blocks 256k chunks

Model: Unknown (unknown)
Disk /dev/md_d0: 15.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
  1      17.4kB  15.0TB  15.0TB               primary


-- LVM2 on top of /dev/md_d0p1

  --- Physical volume ---
   PV Name               /dev/md_d0p1
   VG Name               vgshare
   PV Size               13.62 TB / not usable 3.47 MB
   PE Size (KByte)       4096

   --- Volume group ---
   VG Name               vgshare
   System ID
   Format                lvm2
   VG Size               13.62 TB
   PE Size               4.00 MB

   --- Logical volume ---
   LV Name                /dev/vgshare/share
   VG Name                vgshare
   LV UUID                Aoj27F-kf4U-i6XE-eNWg-hMLX-MS1h-s3oArp
   LV Write Access        read/write
   LV Status              available
   # open                 0
   LV Size                7.00 TB
   Current LE             1835008
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
   - currently set to     2048
   Block device           253:4

-- /dev/vgshare/share is ext4 formated with
mkfs.ext4 -b 4096 -E stride=64,stripe-width=128 /dev/md_d0p1

All well so far. And I start write data...

Then I create a new 2TB test lv
and format it.

/share_7TB is lost, /test_2TB is there....

Mar 12 21:13:28 server kernel: EXT4-fs error (device dm-4): 
ext4_mb_generate_buddy: EXT4-fs: group 0: 32768 blocks in bitmap, 3248 in gd
Mar 12 21:13:30 server kernel: EXT4-fs error (device dm-4): 
ext4_mb_generate_buddy: EXT4-fs: group 1648: 24544 blocks in bitmap, 153 
in gd
Mar 12 21:13:31 server kernel: attempt to access beyond end of device
Mar 12 21:13:31 server kernel: dm-4: rw=2, want=15493450520, 
limit=15032385536
Mar 12 21:13:31 server kernel: attempt to access beyond end of device
--snip
Mar 12 21:17:49 server kernel: EXT4-fs error (device dm-4): 
ext4_mb_release_inode_pa: free 1802, pa_free 1458
Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 93430033 blocks 705745 
reqs (482734 success)
Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 327298 extents scanned, 
241152 goal hits, 219206 2^N hits, 0 breaks, 0 lost
Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 9561 generated and it 
took 2012010656
Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 85047925 preallocated, 
30759591 discarded
Mar 12 21:18:09 server kernel: EXT4-fs: ext4_check_descriptors: Inode 
table for group 0 not in group (block 1936681314)!
Mar 12 21:18:09 server kernel: EXT4-fs: group descriptors corrupted!

GFS instead of ext4 also corrupted.

ext4 on top of /dev/md0 (non partitioned) on top of /dev/sdb /dev/sdc 
(without GPT) also corrupted.

I want to use software raid0 on top of the two hardware radi5
for better performance.

I understood that labeling GPT would solve this problem.
Is it x86 problem? Something fishy with my setup probably
but can't figure it out.

thanks and sorry for long message,
but I can't find a way to get this mirror server up
on it's feet again after this upgrade....

regards,

Giannis

Re: large filesystem corruptions

[Ric Wheeler]

On 03/12/2010 03:01 PM, Kapetanakis Giannis wrote:
> Hi,
>
> The last few days I'm trying to setup a large filesystem
> 14TB and is always getting corrupted...
>
> This is my latest non-working setup. Sorry for long message
> but I want to make clear my  actual setup
>
> 2.6.18-164.11.1.el5PAE (x86)
> 4GB RAM
>
> CONFIG_EFI_VARS=y
> CONFIG_EFI=y
>
> /dev/sdb - 6 x 1.5TB SATA drives in hardware RAID 5 (256 chunk size)
> /dev/sdc - 6 x 1.5TB SATA drives in hardware RAID 5 (256 chunk size)
>
> -- Both hardware raids are GPT labeled
>
> Model: Adaptec ARRAY01 (scsi)
> Disk /dev/sdb: 7489GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
>
> Number  Start   End     Size    File system  Name     Flags
>  1      17.4kB  7489GB  7489GB               ARRAY01
>
> Model: Adaptec ARRAY02 (scsi)
> Disk /dev/sdc: 7489GB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
>
> Number  Start   End     Size    File system  Name     Flags
>  1      17.4kB  7489GB  7489GB               ARRAY02
>
> -- /dev/md_d0 software raid0 on /dev/sdb1 and /dev/sdc1
> -- (doing on /dev/sdb and /dev/sdc also corrupted)
> -- this raid0 is also GPT labeled (256 chunk size)
>
> mdadm --create /dev/md_d0 -a p1 -c 256 -l 0 -n 2 /dev/sdb1 /dev/sdc1
>
>   md_d0 : active raid0 sdb1[0] sdc1[1]
>       14627614208 blocks 256k chunks
>
> Model: Unknown (unknown)
> Disk /dev/md_d0: 15.0TB
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
>
> Number  Start   End     Size    File system  Name     Flags
>  1      17.4kB  15.0TB  15.0TB               primary
>
>
> -- LVM2 on top of /dev/md_d0p1
>
>  --- Physical volume ---
>   PV Name               /dev/md_d0p1
>   VG Name               vgshare
>   PV Size               13.62 TB / not usable 3.47 MB
>   PE Size (KByte)       4096
>
>   --- Volume group ---
>   VG Name               vgshare
>   System ID
>   Format                lvm2
>   VG Size               13.62 TB
>   PE Size               4.00 MB
>
>   --- Logical volume ---
>   LV Name                /dev/vgshare/share
>   VG Name                vgshare
>   LV UUID                Aoj27F-kf4U-i6XE-eNWg-hMLX-MS1h-s3oArp
>   LV Write Access        read/write
>   LV Status              available
>   # open                 0
>   LV Size                7.00 TB
>   Current LE             1835008
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     auto
>   - currently set to     2048
>   Block device           253:4
>
> -- /dev/vgshare/share is ext4 formated with
> mkfs.ext4 -b 4096 -E stride=64,stripe-width=128 /dev/md_d0p1
>
> All well so far. And I start write data...
>
> Then I create a new 2TB test lv
> and format it.
>
> /share_7TB is lost, /test_2TB is there....
>
> Mar 12 21:13:28 server kernel: EXT4-fs error (device dm-4): 
> ext4_mb_generate_buddy: EXT4-fs: group 0: 32768 blocks in bitmap, 3248 
> in gd
> Mar 12 21:13:30 server kernel: EXT4-fs error (device dm-4): 
> ext4_mb_generate_buddy: EXT4-fs: group 1648: 24544 blocks in bitmap, 
> 153 in gd
> Mar 12 21:13:31 server kernel: attempt to access beyond end of device
> Mar 12 21:13:31 server kernel: dm-4: rw=2, want=15493450520, 
> limit=15032385536
> Mar 12 21:13:31 server kernel: attempt to access beyond end of device
> --snip
> Mar 12 21:17:49 server kernel: EXT4-fs error (device dm-4): 
> ext4_mb_release_inode_pa: free 1802, pa_free 1458
> Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 93430033 blocks 
> 705745 reqs (482734 success)
> Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 327298 extents 
> scanned, 241152 goal hits, 219206 2^N hits, 0 breaks, 0 lost
> Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 9561 generated and it 
> took 2012010656
> Mar 12 21:17:49 server kernel: EXT4-fs: mballoc: 85047925 
> preallocated, 30759591 discarded
> Mar 12 21:18:09 server kernel: EXT4-fs: ext4_check_descriptors: Inode 
> table for group 0 not in group (block 1936681314)!
> Mar 12 21:18:09 server kernel: EXT4-fs: group descriptors corrupted!
>
> GFS instead of ext4 also corrupted.
>
> ext4 on top of /dev/md0 (non partitioned) on top of /dev/sdb /dev/sdc 
> (without GPT) also corrupted.
>
> I want to use software raid0 on top of the two hardware radi5
> for better performance.
>
> I understood that labeling GPT would solve this problem.
> Is it x86 problem? Something fishy with my setup probably
> but can't figure it out.
>
> thanks and sorry for long message,
> but I can't find a way to get this mirror server up
> on it's feet again after this upgrade....
>
> regards,
>
> Giannis
>

This is probably an issue with the early version of ext4 you are using - 
note that the support for ext4 > 16TB is still gated by some work done 
up in the tools chain.

Have you tried xfs?

regards,

Ric

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 12/03/10 22:35, Ric Wheeler wrote:

> This is probably an issue with the early version of ext4 you are using -
> note that the support for ext4 > 16TB is still gated by some work done
> up in the tools chain.
>
> Have you tried xfs?
>
> regards,
>
> Ric

Thanks for answering,

My filesystem would be 15 TB < 16 TB.
GFS also crashed and burned so are you sure that this
this is a problem with ext4?
Would Fedora and newer kernel be better?

On my tests the crashing filesystem was 7TB.
When I added a second fs of 2 TB on lvm, so total > 8TB
I had the crash.

I did a new test now and didn't use GFT partitions
but the whole physical/logical drives

sdb -
     | ---> md0 ---> LVM ---> ext4 filesystems
sdc -

all sdb, sdc, md0 are gpt labeled without gpt partitions
inside. No crash so far but without any data written.

Maybe the gpt partitions did the bad thing?
Can md0 use large gpt drives with no partitions?
can lvm2 use large raid device with no partition pv?

I could try XFS but i'm not familiar with it,
so I wouldn't know the optimized values for
such a large fs

regards,

Giannis

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 13/03/10 02:29, Kapetanakis Giannis wrote:
> I did a new test now and didn't use GFT partitions
> but the whole physical/logical drives
>
> sdb -
> | ---> md0 ---> LVM ---> ext4 filesystems
> sdc -
>
> all sdb, sdc, md0 are gpt labeled without gpt partitions
> inside. No crash so far but without any data written.
>
> Maybe the gpt partitions did the bad thing?
> Can md0 use large gpt drives with no partitions?
> can lvm2 use large raid device with no partition pv?

crashed and burned also:

Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): 
ext4_mb_generate_buddy: EXT4-fs: group 48: 24544 blocks in bitmap, 2016 
in gd
Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): 
mb_free_blocks: double-free of inode 12's block 1583104(bit 10240 in 
group 48)
Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): 
mb_free_blocks: double-free of inode 12's block 1583105(bit 10241 in 
group 48)
--snip

so gpt partitions was not a problem.

Next in list: XFS

    682  2:47    mkfs.xfs -f /dev/vgshare/share
    684  2:47    mount /dev/vgshare/share /share/
    686  2:47    mkfs.xfs -f /dev/vgshare/test
    687  2:47    mount /dev/vgshare/test /test/
    689  2:47    cd /share/
    691  2:48    dd if=/dev/zero of=papaki bs=4096

Mar 13 02:47:23 server kernel: Filesystem "dm-4": Disabling barriers, 
not supported by the underlying device
Mar 13 02:47:23 server kernel: XFS mounting filesystem dm-4
Mar 13 02:47:48 server kernel: Filesystem "dm-5": Disabling barriers, 
not supported by the underlying device
Mar 13 02:47:48 server kernel: XFS mounting filesystem dm-5
Mar 13 02:48:05 server kernel: Filesystem "dm-4": XFS internal error 
xfs_trans_cancel at line 1138 of file 
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c. 
  Caller 0xf90e0bbc
Mar 13 02:48:05 server kernel:  [<f90d85fe>] xfs_trans_cancel+0x4d/0xd6 
[xfs]
Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
Mar 13 02:48:05 server kernel:  [<f90e88f4>] xfs_vn_mknod+0x19c/0x380 [xfs]
Mar 13 02:48:05 server kernel:  [<c04760e9>] __getblk+0x30/0x27a
Mar 13 02:48:05 server kernel:  [<f8852ac7>] 
do_get_write_access+0x441/0x46e [jbd]
Mar 13 02:48:05 server kernel:  [<f8889502>] 
__ext3_get_inode_loc+0x109/0x2d5 [ext3]
Mar 13 02:48:05 server kernel:  [<c045a7aa>] 
get_page_from_freelist+0x96/0x370
Mar 13 02:48:05 server kernel:  [<f90b6827>] xfs_dir_lookup+0x91/0xff [xfs]
Mar 13 02:48:05 server kernel:  [<f90c3c51>] xfs_iunlock+0x51/0x6d [xfs]
Mar 13 02:48:05 server kernel:  [<c04824f0>] __link_path_walk+0xc62/0xd33
Mar 13 02:48:05 server kernel:  [<c0480b43>] vfs_create+0xc8/0x12f
Mar 13 02:48:05 server kernel:  [<c04834ef>] open_namei+0x16a/0x5fb
Mar 13 02:48:05 server kernel:  [<c0472a92>] __dentry_open+0xea/0x1ab
Mar 13 02:48:05 server kernel:  [<c0472be2>] do_filp_open+0x1c/0x31
Mar 13 02:48:05 server kernel:  [<c0472c35>] do_sys_open+0x3e/0xae
Mar 13 02:48:05 server kernel:  [<c0472cd2>] sys_open+0x16/0x18
Mar 13 02:48:05 server kernel:  [<c0404f17>] syscall_call+0x7/0xb
Mar 13 02:48:05 server kernel:  =======================
Mar 13 02:48:05 server kernel: xfs_force_shutdown(dm-4,0x8) called from 
line 1139 of file 
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c. 
  Return address = 0xf90eb6c4
Mar 13 02:48:05 server kernel: Filesystem "dm-4": Corruption of 
in-memory data detected.  Shutting down filesystem: dm-4
Mar 13 02:48:05 server kernel: Please umount the filesystem, and rectify 
the problem(s)
Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from 
line 424 of file 
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. 
Return address = 0xf90eb6c4
Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from 
line 424 of file 
/home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. 
Return address = 0xf90eb6c4

xfs_check /dev/vgshare/share
XFS: Log inconsistent (didn't find previous header)
XFS: failed to find log head
ERROR: cannot find log head/tail, run xfs_repair

xfs_repair /dev/vgshare/share
Phase 1 - find and verify superblock...
bad primary superblock - filesystem mkfs-in-progress bit set !!!

attempting to find secondary superblock...
...................................

I stopped it, can't wait to search 7TB to find the secondary 
superblock...probably won't find anything

/test works

So are we sure it's the fs?
Something else is fishy...

regards,

Giannis

Re: large filesystem corruptions

[Michael Evans]

On Fri, Mar 12, 2010 at 4:55 PM, Kapetanakis Giannis
<bilias@edu.physics.uoc.gr> wrote:
> On 13/03/10 02:29, Kapetanakis Giannis wrote:
>>
>> I did a new test now and didn't use GFT partitions
>> but the whole physical/logical drives
>>
>> sdb -
>> | ---> md0 ---> LVM ---> ext4 filesystems
>> sdc -
>>
>> all sdb, sdc, md0 are gpt labeled without gpt partitions
>> inside. No crash so far but without any data written.
>>
>> Maybe the gpt partitions did the bad thing?
>> Can md0 use large gpt drives with no partitions?
>> can lvm2 use large raid device with no partition pv?
>
> crashed and burned also:
>
> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4):
> ext4_mb_generate_buddy: EXT4-fs: group 48: 24544 blocks in bitmap, 2016 in
> gd
> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): mb_free_blocks:
> double-free of inode 12's block 1583104(bit 10240 in group 48)
> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): mb_free_blocks:
> double-free of inode 12's block 1583105(bit 10241 in group 48)
> --snip
>
> so gpt partitions was not a problem.
>
> Next in list: XFS
>
>   682  2:47    mkfs.xfs -f /dev/vgshare/share
>   684  2:47    mount /dev/vgshare/share /share/
>   686  2:47    mkfs.xfs -f /dev/vgshare/test
>   687  2:47    mount /dev/vgshare/test /test/
>   689  2:47    cd /share/
>   691  2:48    dd if=/dev/zero of=papaki bs=4096
>
> Mar 13 02:47:23 server kernel: Filesystem "dm-4": Disabling barriers, not
> supported by the underlying device
> Mar 13 02:47:23 server kernel: XFS mounting filesystem dm-4
> Mar 13 02:47:48 server kernel: Filesystem "dm-5": Disabling barriers, not
> supported by the underlying device
> Mar 13 02:47:48 server kernel: XFS mounting filesystem dm-5
> Mar 13 02:48:05 server kernel: Filesystem "dm-4": XFS internal error
> xfs_trans_cancel at line 1138 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c.
>  Caller 0xf90e0bbc
> Mar 13 02:48:05 server kernel:  [<f90d85fe>] xfs_trans_cancel+0x4d/0xd6
> [xfs]
> Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
> Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
> Mar 13 02:48:05 server kernel:  [<f90e88f4>] xfs_vn_mknod+0x19c/0x380 [xfs]
> Mar 13 02:48:05 server kernel:  [<c04760e9>] __getblk+0x30/0x27a
> Mar 13 02:48:05 server kernel:  [<f8852ac7>] do_get_write_access+0x441/0x46e
> [jbd]
> Mar 13 02:48:05 server kernel:  [<f8889502>]
> __ext3_get_inode_loc+0x109/0x2d5 [ext3]
> Mar 13 02:48:05 server kernel:  [<c045a7aa>]
> get_page_from_freelist+0x96/0x370
> Mar 13 02:48:05 server kernel:  [<f90b6827>] xfs_dir_lookup+0x91/0xff [xfs]
> Mar 13 02:48:05 server kernel:  [<f90c3c51>] xfs_iunlock+0x51/0x6d [xfs]
> Mar 13 02:48:05 server kernel:  [<c04824f0>] __link_path_walk+0xc62/0xd33
> Mar 13 02:48:05 server kernel:  [<c0480b43>] vfs_create+0xc8/0x12f
> Mar 13 02:48:05 server kernel:  [<c04834ef>] open_namei+0x16a/0x5fb
> Mar 13 02:48:05 server kernel:  [<c0472a92>] __dentry_open+0xea/0x1ab
> Mar 13 02:48:05 server kernel:  [<c0472be2>] do_filp_open+0x1c/0x31
> Mar 13 02:48:05 server kernel:  [<c0472c35>] do_sys_open+0x3e/0xae
> Mar 13 02:48:05 server kernel:  [<c0472cd2>] sys_open+0x16/0x18
> Mar 13 02:48:05 server kernel:  [<c0404f17>] syscall_call+0x7/0xb
> Mar 13 02:48:05 server kernel:  =======================
> Mar 13 02:48:05 server kernel: xfs_force_shutdown(dm-4,0x8) called from line
> 1139 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c.
>  Return address = 0xf90eb6c4
> Mar 13 02:48:05 server kernel: Filesystem "dm-4": Corruption of in-memory
> data detected.  Shutting down filesystem: dm-4
> Mar 13 02:48:05 server kernel: Please umount the filesystem, and rectify the
> problem(s)
> Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from line
> 424 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. Return
> address = 0xf90eb6c4
> Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from line
> 424 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. Return
> address = 0xf90eb6c4
>
> xfs_check /dev/vgshare/share
> XFS: Log inconsistent (didn't find previous header)
> XFS: failed to find log head
> ERROR: cannot find log head/tail, run xfs_repair
>
> xfs_repair /dev/vgshare/share
> Phase 1 - find and verify superblock...
> bad primary superblock - filesystem mkfs-in-progress bit set !!!
>
> attempting to find secondary superblock...
> ...................................
>
> I stopped it, can't wait to search 7TB to find the secondary
> superblock...probably won't find anything
>
> /test works
>
> So are we sure it's the fs?
> Something else is fishy...
>
> regards,
>
> Giannis
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

This is a really basic thing, but do you have the x86 support for very
large block devices (I can't remember what the option is, since I've
been running 64 bits on any system that even remotely came close to
needing it anyway) enabled in the config as well?

Here's a hit from google, CONFIG_LBD http://cateee.net/lkddb/web-lkddb/LBD.html

Enable block devices of size 2TB and larger.

Since you're using a device >2TB in size, I will assume you are using
one of the three 'version 1' superblock types.  Either at the end 1.0,
beginning 1.1 or 4kb in from the beginning.

Please provide the full output of mdadm -Dvvs

You can use any block device as a member of an md array.  However if
you are going 'whole drive' then it would be a very good idea to erase
the existing partition table structure prior to putting a raid
superblock on the device.  This way there is no confusion about if the
device has partitions or is in fact a raid member.  Similarly when
transitioning back the other way ensuring that the old metadata for
the array is erased is also a good idea.

The kernel you're running seems to be ... exceptionally old and
heavily patched.  I have no way of knowing if the many, many, patches
that fixed numerous issues over the /years/ since it's release have
been included.  Please make sure you have the most recent release from
your vendor and ask them for support in parallel.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 13/03/10 03:58, Michael Evans wrote:

> This is a really basic thing, but do you have the x86 support for very
> large block devices (I can't remember what the option is, since I've
> been running 64 bits on any system that even remotely came close to
> needing it anyway) enabled in the config as well?
>
> Here's a hit from google, CONFIG_LBD http://cateee.net/lkddb/web-lkddb/LBD.html
>
> Enable block devices of size 2TB and larger.

Yes I have LBD support
grep LBD /boot/config-2.6.18-164.11.1.el5PAE
CONFIG_LBD=y


> Since you're using a device>2TB in size, I will assume you are using
> one of the three 'version 1' superblock types.  Either at the end 1.0,
> beginning 1.1 or 4kb in from the beginning.
>
> Please provide the full output of mdadm -Dvvs

If you mean metadata then I'm at default -> 0.90
Is this the problem? I 've seen in the manual that
2 TB is the limit for raid 1 and above

"0, 0.90, default:   Use the original 0.90  format  superblock.   This 
format limits  arrays  to 28 component devices and limits component 
devices of levels 1 and greater to 2 terabytes."


[root@server ~]# mdadm -Dvvs
mdadm: bad uuid: UUID=324587ca:484d94c7:f06cbaee:5b63cd3
/dev/md0:
         Version : 0.90
   Creation Time : Sat Mar 13 02:00:23 2010
      Raid Level : raid0
      Array Size : 14627614208 (13949.98 GiB 14978.68 GB)
    Raid Devices : 2
   Total Devices : 2
Preferred Minor : 0
     Persistence : Superblock is persistent

     Update Time : Sat Mar 13 02:00:23 2010
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0

      Chunk Size : 256K

            UUID : 324587ca:484d94c7:f06cbaee:5b63cd37
          Events : 0.1

     Number   Major   Minor   RaidDevice State
        0       8       16        0      active sync   /dev/sdb
        1       8       32        1      active sync   /dev/sdc



> You can use any block device as a member of an md array.  However if
> you are going 'whole drive' then it would be a very good idea to erase
> the existing partition table structure prior to putting a raid
> superblock on the device.  This way there is no confusion about if the
> device has partitions or is in fact a raid member.  Similarly when
> transitioning back the other way ensuring that the old metadata for
> the array is erased is also a good idea.

I have erased prior creating the gpt and the raid devide
dd if=/dev/zero of=/dev/sdb bs=512 count=64
dd if=/dev/zero of=/dev/sdc bs=512 count=64

(I accidentally erased my boot disk also but I managed to recover)

> The kernel you're running seems to be ... exceptionally old and
> heavily patched.  I have no way of knowing if the many, many, patches
> that fixed numerous issues over the /years/ since it's release have
> been included.  Please make sure you have the most recent release from
> your vendor and ask them for support in parallel.

This is centos 5.4 stock kernel. So this is redhat 5.4 stock kernel
They say they support all these...

thanks
Giannis

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 13/03/10 03:58, Michael Evans wrote:

> Since you're using a device>2TB in size, I will assume you are using
> one of the three 'version 1' superblock types.  Either at the end 1.0,
> beginning 1.1 or 4kb in from the beginning.
>
> Please provide the full output of mdadm -Dvvs

I did a test with 1.1 metadata version.
Crashed again when I deleted the file from the large FS.

I did a mistake in my previous mail.
Kernel is not stock. It's the latest kernel from centos/redhat 5.4
System up2date.

I will open a ticket in Centos and Redhat bugzilla
and see how it goes.

Giannis
ps. Probably install fedora 12 64bit on Monday

Re: large filesystem corruptions

[Ric Wheeler]

On 03/12/2010 08:58 PM, Michael Evans wrote:
> On Fri, Mar 12, 2010 at 4:55 PM, Kapetanakis Giannis
> <bilias@edu.physics.uoc.gr>  wrote:
>    
>> On 13/03/10 02:29, Kapetanakis Giannis wrote:
>>      
>>> I did a new test now and didn't use GFT partitions
>>> but the whole physical/logical drives
>>>
>>> sdb -
>>> | --->  md0 --->  LVM --->  ext4 filesystems
>>> sdc -
>>>
>>> all sdb, sdc, md0 are gpt labeled without gpt partitions
>>> inside. No crash so far but without any data written.
>>>
>>> Maybe the gpt partitions did the bad thing?
>>> Can md0 use large gpt drives with no partitions?
>>> can lvm2 use large raid device with no partition pv?
>>>        
>> crashed and burned also:
>>
>> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4):
>> ext4_mb_generate_buddy: EXT4-fs: group 48: 24544 blocks in bitmap, 2016 in
>> gd
>> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): mb_free_blocks:
>> double-free of inode 12's block 1583104(bit 10240 in group 48)
>> Mar 13 02:40:28 server kernel: EXT4-fs error (device dm-4): mb_free_blocks:
>> double-free of inode 12's block 1583105(bit 10241 in group 48)
>> --snip
>>
>> so gpt partitions was not a problem.
>>
>> Next in list: XFS
>>
>>    682  2:47    mkfs.xfs -f /dev/vgshare/share
>>    684  2:47    mount /dev/vgshare/share /share/
>>    686  2:47    mkfs.xfs -f /dev/vgshare/test
>>    687  2:47    mount /dev/vgshare/test /test/
>>    689  2:47    cd /share/
>>    691  2:48    dd if=/dev/zero of=papaki bs=4096
>>
>> Mar 13 02:47:23 server kernel: Filesystem "dm-4": Disabling barriers, not
>> supported by the underlying device
>> Mar 13 02:47:23 server kernel: XFS mounting filesystem dm-4
>> Mar 13 02:47:48 server kernel: Filesystem "dm-5": Disabling barriers, not
>> supported by the underlying device
>> Mar 13 02:47:48 server kernel: XFS mounting filesystem dm-5
>> Mar 13 02:48:05 server kernel: Filesystem "dm-4": XFS internal error
>> xfs_trans_cancel at line 1138 of file
>> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c.
>>   Caller 0xf90e0bbc
>> Mar 13 02:48:05 server kernel:  [<f90d85fe>] xfs_trans_cancel+0x4d/0xd6
>> [xfs]
>> Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
>> Mar 13 02:48:05 server kernel:  [<f90e0bbc>] xfs_create+0x4ec/0x525 [xfs]
>> Mar 13 02:48:05 server kernel:  [<f90e88f4>] xfs_vn_mknod+0x19c/0x380 [xfs]
>> Mar 13 02:48:05 server kernel:  [<c04760e9>] __getblk+0x30/0x27a
>> Mar 13 02:48:05 server kernel:  [<f8852ac7>] do_get_write_access+0x441/0x46e
>> [jbd]
>> Mar 13 02:48:05 server kernel:  [<f8889502>]
>> __ext3_get_inode_loc+0x109/0x2d5 [ext3]
>> Mar 13 02:48:05 server kernel:  [<c045a7aa>]
>> get_page_from_freelist+0x96/0x370
>> Mar 13 02:48:05 server kernel:  [<f90b6827>] xfs_dir_lookup+0x91/0xff [xfs]
>> Mar 13 02:48:05 server kernel:  [<f90c3c51>] xfs_iunlock+0x51/0x6d [xfs]
>> Mar 13 02:48:05 server kernel:  [<c04824f0>] __link_path_walk+0xc62/0xd33
>> Mar 13 02:48:05 server kernel:  [<c0480b43>] vfs_create+0xc8/0x12f
>> Mar 13 02:48:05 server kernel:  [<c04834ef>] open_namei+0x16a/0x5fb
>> Mar 13 02:48:05 server kernel:  [<c0472a92>] __dentry_open+0xea/0x1ab
>> Mar 13 02:48:05 server kernel:  [<c0472be2>] do_filp_open+0x1c/0x31
>> Mar 13 02:48:05 server kernel:  [<c0472c35>] do_sys_open+0x3e/0xae
>> Mar 13 02:48:05 server kernel:  [<c0472cd2>] sys_open+0x16/0x18
>> Mar 13 02:48:05 server kernel:  [<c0404f17>] syscall_call+0x7/0xb
>> Mar 13 02:48:05 server kernel:  =======================
>> Mar 13 02:48:05 server kernel: xfs_force_shutdown(dm-4,0x8) called from line
>> 1139 of file
>> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_trans.c.
>>   Return address = 0xf90eb6c4
>> Mar 13 02:48:05 server kernel: Filesystem "dm-4": Corruption of in-memory
>> data detected.  Shutting down filesystem: dm-4
>> Mar 13 02:48:05 server kernel: Please umount the filesystem, and rectify the
>> problem(s)
>> Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from line
>> 424 of file
>> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. Return
>> address = 0xf90eb6c4
>> Mar 13 02:48:45 server kernel: xfs_force_shutdown(dm-4,0x1) called from line
>> 424 of file
>> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_PAE/xfs_rw.c. Return
>> address = 0xf90eb6c4
>>
>> xfs_check /dev/vgshare/share
>> XFS: Log inconsistent (didn't find previous header)
>> XFS: failed to find log head
>> ERROR: cannot find log head/tail, run xfs_repair
>>
>> xfs_repair /dev/vgshare/share
>> Phase 1 - find and verify superblock...
>> bad primary superblock - filesystem mkfs-in-progress bit set !!!
>>
>> attempting to find secondary superblock...
>> ...................................
>>
>> I stopped it, can't wait to search 7TB to find the secondary
>> superblock...probably won't find anything
>>
>> /test works
>>
>> So are we sure it's the fs?
>> Something else is fishy...
>>
>> regards,
>>
>> Giannis
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>      
> This is a really basic thing, but do you have the x86 support for very
> large block devices (I can't remember what the option is, since I've
> been running 64 bits on any system that even remotely came close to
> needing it anyway) enabled in the config as well?
>
> Here's a hit from google, CONFIG_LBD http://cateee.net/lkddb/web-lkddb/LBD.html
>
> Enable block devices of size 2TB and larger.
>
> Since you're using a device>2TB in size, I will assume you are using
> one of the three 'version 1' superblock types.  Either at the end 1.0,
> beginning 1.1 or 4kb in from the beginning.
>
> Please provide the full output of mdadm -Dvvs
>
> You can use any block device as a member of an md array.  However if
> you are going 'whole drive' then it would be a very good idea to erase
> the existing partition table structure prior to putting a raid
> superblock on the device.  This way there is no confusion about if the
> device has partitions or is in fact a raid member.  Similarly when
> transitioning back the other way ensuring that the old metadata for
> the array is erased is also a good idea.
>
> The kernel you're running seems to be ... exceptionally old and
> heavily patched.  I have no way of knowing if the many, many, patches
> that fixed numerous issues over the /years/ since it's release have
> been included.  Please make sure you have the most recent release from
> your vendor and ask them for support in parallel.
>    

I would agree that it would be key to try this on a newer kernel & on a 
64 bit box. If you have an issue with a specific vendor release, you 
should open a ticket/bugzilla with that vendor so they can help you 
figure this out.

ric

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 13/03/10 15:07, Ric Wheeler wrote:
> I would agree that it would be key to try this on a newer kernel & on 
> a 64 bit box. If you have an issue with a specific vendor release, you 
> should open a ticket/bugzilla with that vendor so they can help you 
> figure this out.
>
> ric

I will try Fedora 12/64bit on Monday when I have physical access.

Right now all tests have failed.
OS in Centos 5.4 x86 PAE so I opened a ticket both in Centos and Redhat 
bugzilla.
I don't have any contract with redhat but I hope they will give a try on 
this.

https://bugzilla.redhat.com/show_bug.cgi?id=573185
http://bugs.centos.org/view.php?id=4239

My feelings are that it has to do with the software raid layer and not
the filesystem (ext4, xfs, gfs all died).
The filesystem seems to not get the appropriate physical/logical
quandaries from the raid layer.

Unless the x86 kernel is not capable of addressing
large filesystems as it is documented (16TB as you said).

best regards,

Giannis

Re: large filesystem corruptions

[Bill Davidsen]

Kapetanakis Giannis wrote:
> On 12/03/10 22:35, Ric Wheeler wrote:
>
>> This is probably an issue with the early version of ext4 you are using -
>> note that the support for ext4 > 16TB is still gated by some work done
>> up in the tools chain.
>>
>> Have you tried xfs?
>>
>> regards,
>>
>> Ric
>
> Thanks for answering,
>
> My filesystem would be 15 TB < 16 TB.
> GFS also crashed and burned so are you sure that this
> this is a problem with ext4?
> Would Fedora and newer kernel be better?
>
> On my tests the crashing filesystem was 7TB.
> When I added a second fs of 2 TB on lvm, so total > 8TB
> I had the crash.
>
> I did a new test now and didn't use GFT partitions
> but the whole physical/logical drives
>
> sdb -
>     | ---> md0 ---> LVM ---> ext4 filesystems
> sdc -
>
> all sdb, sdc, md0 are gpt labeled without gpt partitions
> inside. No crash so far but without any data written.
>
> Maybe the gpt partitions did the bad thing?
> Can md0 use large gpt drives with no partitions?
> can lvm2 use large raid device with no partition pv?

I did not notice the LVM on your first post, it's far enough down that I 
missed it. I would be suspicious that the problem is in LVM rather than 
md, but if you're going for a new install soon it's probably not worth 
checking.

If you have video issues with FC12 you may need to run the vesa driver 
as a quick solution. There have been learning experiences in the video 
drivers in recent kernels, not just Fedora.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 14/03/10 01:45, Bill Davidsen wrote:

> I did not notice the LVM on your first post, it's far enough down that I
> missed it. I would be suspicious that the problem is in LVM rather than
> md, but if you're going for a new install soon it's probably not worth
> checking.
>
> If you have video issues with FC12 you may need to run the vesa driver
> as a quick solution. There have been learning experiences in the video
> drivers in recent kernels, not just Fedora.

No it's not the LVM layer.
I have crashes without LVM as well.

gpt label is also corrupted, so my guesses go
to raid layer (that's why i posted here in the first place)
or maybe the driver-adaptec.

Giannis

Re: large filesystem corruptions

[Kapetanakis Giannis]

On 13/03/10 15:19, Kapetanakis Giannis wrote:

> I will try Fedora 12/64bit on Monday when I have physical access.
>
> Right now all tests have failed.
> OS in Centos 5.4 x86 PAE so I opened a ticket both in Centos and Redhat
> bugzilla.
> I don't have any contract with redhat but I hope they will give a try on
> this.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=573185
> http://bugs.centos.org/view.php?id=4239
>
> My feelings are that it has to do with the software raid layer and not
> the filesystem (ext4, xfs, gfs all died).
> The filesystem seems to not get the appropriate physical/logical
> quandaries from the raid layer.
>
> Unless the x86 kernel is not capable of addressing
> large filesystems as it is documented (16TB as you said).

I thought I should report here as well.

I've installed Centos 5.4 x64 and all going well so far.
No corruptions at all.

My setup is as before

sdb (hard raid5)
  |
  |------> md0 (soft raid0)----> LVM ----> ext4
  |
sdc (hard raid5)

The only strange thing I've noticed is that pvcreate
is messing with the gpt label on md0. Don't know of that
is normal.

https://bugzilla.redhat.com/show_bug.cgi?id=573185#c4

#fdisk -l /dev/md0

WARNING: GPT (GUID Partition Table) detected on '/dev/md0'! The util fdisk
doesn't support GPT. Use GNU Parted.


Disk /dev/md0: 14978.6 GB, 14978676948992 bytes
2 heads, 4 sectors/track, -638063744 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md0 doesn't contain a valid partition table

# parted /dev/md0
GNU Parted 1.8.1
Using /dev/md0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: Unable to open /dev/md0 - unrecognised disk label.

thanks all

Giannis