64k Page size + ext3 errors

64k Page size + ext3 errors

[tirumalareddy marri]

I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs errors when I use 64K page size. When I use 4K page size I don't see any issue. As many of you know, we will get better performance when we store bigger files like videos.
 
When I copy 128MB size files using 64k page size no issues seen. When I tried to copy 1.8 GB file with 64KB page size support I am seeing the following errors. Any clue what could be wrong. 
 
Errors 1:
EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65533, length 1
EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65534, length 1
 
 
Errors2:
EXT3-fs error (device md0): ext3_readdir: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=3040

 
Thanks in Advance,
Marri
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 64k Page size + ext3 errors

[sandeen]

> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs
> errors when I use 64K page size. When I use 4K page size I don't see any
> issue. As many of you know, we will get better performance when we store
> bigger files like videos.

Do you think it is specific to the filesystem on raid-5?  Have you tested
w/o md?  I'd be a little surprised if the problem is at the raid layer (so
cc'ing the ext3/4 list), but maybe you tested that already.

Which arch is it, and which kernel?  How big is the filesystem?  With full
geometry information it might be easier for someone to reproduce & debug.

Thanks,
-Eric

> When I copy 128MB size files using 64k page size no issues seen. When I
> tried to copy 1.8 GB file with 64KB page size support I am seeing the
> following errors. Any clue what could be wrong.
>  
> Errors 1:
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system
> zone - blocks from 65533, length 1
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system
> zone - blocks from 65534, length 1
>  
>  
> Errors2:
> EXT3-fs error (device md0): ext3_readdir: bad entry in directory #2:
> rec_len % 4 != 0 - offset=0, inode=3040
>
>  
> Thanks in Advance,
> Marri

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 64k Page size + ext3 errors

[Dan Williams]

On Sun, Jul 27, 2008 at 6:52 PM,  <sandeen@sandeen.net> wrote:
>> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs
>> errors when I use 64K page size. When I use 4K page size I don't see any
>> issue. As many of you know, we will get better performance when we store
>> bigger files like videos.
>
> Do you think it is specific to the filesystem on raid-5?  Have you tested
> w/o md?  I'd be a little surprised if the problem is at the raid layer (so
> cc'ing the ext3/4 list), but maybe you tested that already.
>
> Which arch is it, and which kernel?  How big is the filesystem?  With full
> geometry information it might be easier for someone to reproduce & debug.
>

Marri, can you post your XOR driver the problem may be there.

Thanks,
Dan

Re: 64k Page size + ext3 errors

[Roger Heflin]

tirumalareddy marri wrote:
> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs errors when I use 64K page size. When I use 4K page size I don't see any issue. As many of you know, we will get better performance when we store bigger files like videos.
>  
> When I copy 128MB size files using 64k page size no issues seen. When I tried to copy 1.8 GB file with 64KB page size support I am seeing the following errors. Any clue what could be wrong. 
>  
> Errors 1:
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65533, length 1
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65534, length 1
>  
>  
> Errors2:
> EXT3-fs error (device md0): ext3_readdir: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=3040
> 


All that error means is that something screwed up the filesystem stuff when you 
copied the large file.

The lack of an error in the first case does not mean that the correct stuff was 
written to the filesystem, just that nothing screwed up the internal filesystem 
data, or that the cache saved you.

I would suggest setting up a simple test using no filesystem and all, and make 
sure that the correct data can be read and written (and a large enough amount of 
data that you are not reading out of cache) from the MD device directly, write a 
  specific pattern that would have lots of unique data.   If you don't do enough 
data then things *WILL* be coming from cache and still could be screwed up on 
disk, and this may be what is going on in the case of the 128MB  vs 1.8GB case, 
in both cases it may be wrong on disk, but with the 128MB case is coming from 
cache, and in the 1.8GB case is coming off disk.

                                  Roger

Re: 64k Page size + ext3 errors

[tirumalareddy marri]

Hi Eric,
    This is PowerPC architecture from AMCC(I work for AMCC). I am using 2.6.25 Kernel. According your suggestion I tested with out RAID I did not see any EXT-3 errors. I use "sum" command to check the data integrity of the files. 
I ported ADMA driver to do HW accelerated XOR calculations . Also please note that RAID-5 works with 4K Page size.
Thanks,
Marri


----- Original Message ----
From: "sandeen@sandeen.net" <sandeen@sandeen.net>
To: tirumalareddy marri <tirumalareddymarri@yahoo.com>
Cc: linux-raid@vger.kernel.org; linux-ext4@vger.kernel.org
Sent: Sunday, July 27, 2008 6:52:05 PM
Subject: Re: 64k Page size + ext3 errors

> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs
> errors when I use 64K page size. When I use 4K page size I don't see any
> issue. As many of you know, we will get better performance when we store
> bigger files like videos.

Do you think it is specific to the filesystem on raid-5?  Have you tested
w/o md?  I'd be a little surprised if the problem is at the raid layer (so
cc'ing the ext3/4 list), but maybe you tested that already.

Which arch is it, and which kernel?  How big is the filesystem?  With full
geometry information it might be easier for someone to reproduce & debug.

Thanks,
-Eric

> When I copy 128MB size files using 64k page size no issues seen. When I
> tried to copy 1.8 GB file with 64KB page size support I am seeing the
> following errors. Any clue what could be wrong.
>  
> Errors 1:
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system
> zone - blocks from 65533, length 1
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system
> zone - blocks from 65534, length 1
>  
>  
> Errors2:
> EXT3-fs error (device md0): ext3_readdir: bad entry in directory #2:
> rec_len % 4 != 0 - offset=0, inode=3040
>
>  
> Thanks in Advance,
> Marri
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 64k Page size + ext3 errors

[tirumalareddy marri]

Hi Roger,
   I did sync after I copied the 128MB data. Isn't that should guarantee data is flushed to disk ? I am using "sum" command to check if data file is copied to Disk is valid or not. 
Here is more information.
setup: Created /dev/md0 of 30GB size , created ext3 files system. Then started SAMBA server to export mountded /dev/md0 to a windows machine to run IO and copy files.
4K Page size:
-------------------
1. IO Meter Test: Works just fine.
2. Copied 1.8 GB file and check sum is good.
3. Performance is not good because of small page size.
16k Page size:
---------------------
1. RAID-5 fails some times with " Attempt to access beyond the end of device"
2. Copied 128MB and 385MB file. Checked check sum, they are matching check sum.
3. Copied 1.8 GB file , this failed checksum test using "sum" command. I see "EXT3-fs errors".
64K Page size:
----------------------
1. RAID-5 failes some times with "Attempt to access beyond the end of device"
2. Able to copy 128MB data and check sum test passed.
3. Copying 385MB and 1.8 GB file with EXT3-fs errors.
Thanks,
Marri


----- Original Message ----
From: Roger Heflin <rogerheflin@gmail.com>
To: tirumalareddy marri <tirumalareddymarri@yahoo.com>
Cc: linux-raid@vger.kernel.org
Sent: Sunday, July 27, 2008 7:10:07 PM
Subject: Re: 64k Page size + ext3 errors

tirumalareddy marri wrote:
> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs errors when I use 64K page size. When I use 4K page size I don't see any issue. As many of you know, we will get better performance when we store bigger files like videos.
>  
> When I copy 128MB size files using 64k page size no issues seen. When I tried to copy 1.8 GB file with 64KB page size support I am seeing the following errors. Any clue what could be wrong. 
>  
> Errors 1:
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65533, length 1
> EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - blocks from 65534, length 1
>  
>  
> Errors2:
> EXT3-fs error (device md0): ext3_readdir: bad entry in directory #2: rec_len % 4 != 0 - offset=0, inode=3040
> 


All that error means is that something screwed up the filesystem stuff when you 
copied the large file.

The lack of an error in the first case does not mean that the correct stuff was 
written to the filesystem, just that nothing screwed up the internal filesystem 
data, or that the cache saved you.

I would suggest setting up a simple test using no filesystem and all, and make 
sure that the correct data can be read and written (and a large enough amount of 
data that you are not reading out of cache) from the MD device directly, write a 
  specific pattern that would have lots of unique data.  If you don't do enough 
data then things *WILL* be coming from cache and still could be screwed up on 
disk, and this may be what is going on in the case of the 128MB  vs 1.8GB case, 
in both cases it may be wrong on disk, but with the 128MB case is coming from 
cache, and in the 1.8GB case is coming off disk.

                                  Roger
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 64k Page size + ext3 errors

[Roger Heflin]

tirumalareddy marri wrote:
> Hi Roger,
>    I did sync after I copied the 128MB data. Isn't that should guarantee data is flushed to disk ? I am using "sum" command to check if data file is copied to Disk is valid or not. 

It means it will be flushed to disk, it does not mean that when you read it back 
that will come off disk, if it is still in memory then it will come out of 
memory, and still be wrong on disk.    If you won't want to to more complicated 
test it might be best to create the file, csum it and if it is ok umount the 
device and remount it and csum it again and see, this should at least force it 
to come off of disk again.

How much memory does your test machine have?


> Here is more information.
> setup: Created /dev/md0 of 30GB size , created ext3 files system. Then started SAMBA server to export mountded /dev/md0 to a windows machine to run IO and copy files.
> 4K Page size:
> -------------------
> 1. IO Meter Test: Works just fine.

None of the benchmarks I am familiar with actually confirm that the data is 
good, the only way one of the benchmarks will fail is if the file table gets 
corrupted, and they may run in cache.

> 2. Copied 1.8 GB file and check sum is good.
> 3. Performance is not good because of small page size.
> 16k Page size:
> ---------------------
> 1. RAID-5 fails some times with " Attempt to access beyond the end of device"
> 2. Copied 128MB and 385MB file. Checked check sum, they are matching check sum.
> 3. Copied 1.8 GB file , this failed checksum test using "sum" command. I see "EXT3-fs errors".
> 64K Page size:
> ----------------------
> 1. RAID-5 failes some times with "Attempt to access beyond the end of device"
> 2. Able to copy 128MB data and check sum test passed.
> 3. Copying 385MB and 1.8 GB file with EXT3-fs errors.
> Thanks,
> Marri

I would write directly to the /dev/mdx a specific pattern (a stream of binary 
numbers from 1 ... whatever works fine), and then read that back and see how 
things match or don't.   csum *can* fail, and if you have enough memory then any 
corruption actually on disk *WON'T* be found until somethings causes it to be 
ejected from cache, and then later re-read from disk.

                               Roger

Re: 64k Page size + ext3 errors

[tirumalareddy marri]

Dan,
  Sure  I will post my driver with some cleanup done.
Thanks,
Marri

----- Original Message ----
From: Dan Williams <dan.j.williams@intel.com>
To: sandeen@sandeen.net
Cc: tirumalareddy marri <tirumalareddymarri@yahoo.com>; linux-raid@vger.kernel.org; linux-ext4@vger.kernel.org
Sent: Monday, July 28, 2008 1:04:39 AM
Subject: Re: 64k Page size + ext3 errors

On Sun, Jul 27, 2008 at 6:52 PM,  <sandeen@sandeen.net> wrote:
>> I am using HW accelerated XOR engine for RAID-5 . I am seeing EXT3-fs
>> errors when I use 64K page size. When I use 4K page size I don't see any
>> issue. As many of you know, we will get better performance when we store
>> bigger files like videos.
>
> Do you think it is specific to the filesystem on raid-5?  Have you tested
> w/o md?  I'd be a little surprised if the problem is at the raid layer (so
> cc'ing the ext3/4 list), but maybe you tested that already.
>
> Which arch is it, and which kernel?  How big is the filesystem?  With full
> geometry information it might be easier for someone to reproduce & debug.
>

Marri, can you post your XOR driver the problem may be there.

Thanks,
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html