Attempt to Access Beyond End of Device

Attempt to Access Beyond End of Device

[Federico Sevilla III]

[-- Attachment #1: Type: text/plain, Size: 2349 bytes --]

Hi,

We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine with
two 73.4GB SAS hard drives in hardware RAID 1 with a battery-backed
cache. We are using the stock Debian 2.6.18-4-686 kernel.

The filesystems were created using the following optimization:

        -l size=32768b,version=2 -n 64k

The filesystems are mounted using the following optimization:

        logbufs=8,logbsize=256k

Today, we were unable to mount the "large" (not really at < 70GB) /var
(/dev/sda8), getting the following error:

        Attempt to access beyond end of device
        sda: rw=0, want=143139140, limit=143134720
        I/O error in filesystem ("sda8") meta-data dev sda8 block
        0x75cd27f
        ("xf:read_buf") errors buf count 512
        XFS: size check 2 failed
        Mount: /dev/sda8: can't read superblock

An attempted repair also fails:

        # xfs_repair /dev/sda8
        Phase 1 - find and verify superblock...
        Attempt to access beyond end of device
        Sda: rw=0, want=143139140, limit=143134720
        Xfs_repair: read failed.      Input/output error

The partition table looks okay:

        #Partition table of /dev/sda
        /dev/sda1 : start=          63,size        9637, Id=83, bootable
        /dev/sda2 : start=       96390,size   143042760, Id=5
        /dev/sda3 : start=           0,size           0, Id=0
        /dev/sda4 : start=           0,size           0, Id=0   
        /dev/sda5 : start=       96453,size     3903732, Id=82
        /dev/sda6 : start=     4000248,size     7807527, Id=83
        /dev/sda7 : start=    11807838,size     7807527, Id=83
        /dev/sda8 : start=    19615428,size   123523722, Id=83

The machine doesn't have valuable data, yet, so a simple reinstall
should help get it back up. However I'm more concerned about what could
cause this. It's the first time for me to use a version 2 log, 64k
directories (?) and 256k in-memory log buffers. Are any of these to
blame?

We're also looking for generic filesystem tweaks for a PostgreSQL +
Apache server, which this will be (with more load direct to PostgreSQL
than to/through Apache). Are the above choices for mkfs.xfs and mount
well-made?

Please advise.

Thank you very much.

-- 
Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Attempt to Access Beyond End of Device

[Justin Piszcz]

On Mon, 10 Sep 2007, Federico Sevilla III wrote:

> Hi,
>
> We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine with
> two 73.4GB SAS hard drives in hardware RAID 1 with a battery-backed
> cache. We are using the stock Debian 2.6.18-4-686 kernel.
>
> The filesystems were created using the following optimization:
>
>        -l size=32768b,version=2 -n 64k
>
> The filesystems are mounted using the following optimization:
>
>        logbufs=8,logbsize=256k
>
> Today, we were unable to mount the "large" (not really at < 70GB) /var
> (/dev/sda8), getting the following error:
>
>        Attempt to access beyond end of device
>        sda: rw=0, want=143139140, limit=143134720
>        I/O error in filesystem ("sda8") meta-data dev sda8 block
>        0x75cd27f
>        ("xf:read_buf") errors buf count 512
>        XFS: size check 2 failed
>        Mount: /dev/sda8: can't read superblock
>
> An attempted repair also fails:
>
>        # xfs_repair /dev/sda8
>        Phase 1 - find and verify superblock...
>        Attempt to access beyond end of device
>        Sda: rw=0, want=143139140, limit=143134720
>        Xfs_repair: read failed.      Input/output error
>
> The partition table looks okay:
>
>        #Partition table of /dev/sda
>        /dev/sda1 : start=          63,size        9637, Id=83, bootable
>        /dev/sda2 : start=       96390,size   143042760, Id=5
>        /dev/sda3 : start=           0,size           0, Id=0
>        /dev/sda4 : start=           0,size           0, Id=0
>        /dev/sda5 : start=       96453,size     3903732, Id=82
>        /dev/sda6 : start=     4000248,size     7807527, Id=83
>        /dev/sda7 : start=    11807838,size     7807527, Id=83
>        /dev/sda8 : start=    19615428,size   123523722, Id=83
>
> The machine doesn't have valuable data, yet, so a simple reinstall
> should help get it back up. However I'm more concerned about what could
> cause this. It's the first time for me to use a version 2 log, 64k
> directories (?) and 256k in-memory log buffers. Are any of these to
> blame?
>
> We're also looking for generic filesystem tweaks for a PostgreSQL +
> Apache server, which this will be (with more load direct to PostgreSQL
> than to/through Apache). Are the above choices for mkfs.xfs and mount
> well-made?
>
> Please advise.
>
> Thank you very much.
>
> -- 
> Federico Sevilla III
> F S 3 Consulting Inc.
> http://www.fs3.ph
>

I believe the logbsize should be in bytes, so 262144, it may also take
256k though I have not tried it.

As far as the creation, that's a good question, I use SW raid so XFS
auto-tunes it for my hardware.

That looks mighty strange, what kind of raid card?

Justin.

Re: Attempt to Access Beyond End of Device

[Federico Sevilla III]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

On Mon, 2007-09-10 at 10:44 -0400, Justin Piszcz wrote:
> I believe the logbsize should be in bytes, so 262144, it may also take
> 256k though I have not tried it.

It accepts 256k. The initial mount succeeded, and the problem did not
surface until after some reboot (ie: not the first reboot, either, that
went fine).

> That looks mighty strange, what kind of raid card?

This is the built-in Adaptec AACRAID of the IBM x3400, doing a pretty
simplistic RAID 1.

-- 
Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Attempt to Access Beyond End of Device

[Justin Piszcz]

On Mon, 10 Sep 2007, Federico Sevilla III wrote:

> On Mon, 2007-09-10 at 10:44 -0400, Justin Piszcz wrote:
>> I believe the logbsize should be in bytes, so 262144, it may also take
>> 256k though I have not tried it.
>
> It accepts 256k. The initial mount succeeded, and the problem did not
> surface until after some reboot (ie: not the first reboot, either, that
> went fine).
>
>> That looks mighty strange, what kind of raid card?
>
> This is the built-in Adaptec AACRAID of the IBM x3400, doing a pretty
> simplistic RAID 1.
>
> -- 
> Federico Sevilla III
> F S 3 Consulting Inc.
> http://www.fs3.ph
>

Have you checked the following page?

http://linuxmafia.com/faq/Hardware/sata.html

Does the error occur if you use a different filesystem?

Justin.

Re: Attempt to Access Beyond End of Device

[Eric Sandeen]

Federico Sevilla III wrote:
> Hi,
> 
> We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine with
> two 73.4GB SAS hard drives in hardware RAID 1 with a battery-backed
> cache. We are using the stock Debian 2.6.18-4-686 kernel.

can you send along the results of:

	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?

and
	# cat /proc/partitions

... and does the hardware raid have a funky sector size?

-Eric

Re: Attempt to Access Beyond End of Device

[Federico Sevilla III]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

On Mon, 2007-09-10 at 11:45 -0400, Justin Piszcz wrote:
> Have you checked the following page?
> 
> http://linuxmafia.com/faq/Hardware/sata.html

I checked it now (thanks!) and I believe this is what we've got (or
similar) as we use AACRAID:

        Adaptec AAR 2400, 2410, 2410SA, 2120S, 2200S, 2810SA (8-port),
        21610SA (16-port) series PCI cards — real hardware RAID, using
        the slightly anemic Intel IOP302/303 I/O co-processor chips. Use
        "aacraid" driver. (Should not be confused with the Adaptec 2400A
        ATA RAID host adapter, for which one uses the dpt_i2o driver,
        that card being a legacy of Adaptec's buying DPT — nor with the
        low-end Adaptec AAR 12x0 series, which please see.) Faster at
        random I/O than the 3Ware cards. Optional battery is available
        for the card's cache, for more reliable operation in the event
        of power loss, etc. (Card disables the drive's write cache.)

We also have the optional battery mentioned.

> Does the error occur if you use a different filesystem?

I haven't tried any other filesystem, yet. I'm looking for clues as to
what this could be pointing to, first... and emails like yours are
definitely helping, thank you very much.

Cheers!

-- 
Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Attempt to Access Beyond End of Device

[Federico Sevilla III]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Mon, 2007-09-10 at 10:47 -0500, Eric Sandeen wrote:
> can you send along the results of:
> 
> 	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?

Will do first thing tomorrow when I get back access to the machine, and
will revert to the list.

> and
> 	# cat /proc/partitions
> 
> ... and does the hardware raid have a funky sector size?

Please advise as to how best to usually find this out. For whatever it's
worth, it doesn't allow us to pick a stripe width for RAID 1 (as we
probably shouldn't be able to specify one, anyway). So whatever we're
using is our only choice as far as the hardware RAID 1 goes. (We could
probably use software RAID 1, though, if push comes to shove.)

Thank you very much for your time.

-- 
Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Attempt to Access Beyond End of Device

[Eric Sandeen]

Federico Sevilla III wrote:
> On Mon, 2007-09-10 at 10:47 -0500, Eric Sandeen wrote:
>> can you send along the results of:
>>
>> 	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?
> 
> Will do first thing tomorrow when I get back access to the machine, and
> will revert to the list.
> 
>> and
>> 	# cat /proc/partitions
>>
>> ... and does the hardware raid have a funky sector size?
> 
> Please advise as to how best to usually find this out. 

Dunno :)

FWIW, the size check that is failing is simply looking at the fs
geometry and trying to go out and read the last sector.

-Eric

Re: Attempt to Access Beyond End of Device

[Federico Sevilla III]

Hi,

Quoting Bhagi rathi <jahnu77@gmail.com>:
>  I hope your problem with xfs_repair is resolved.

It wasn't an XFS-centric problem, after all. Running cfdisk on the  
system would likewise fail because the partition had been set beyond  
the disk boundary. I'm not sure about how this happened. Either the  
controller reported a large size to Debian during installation, or  
Debian mis-read the reported size on installation. At any rate,  
redoing the entire installation (including RAID setup) with the same  
procedure resulted in the correct (ie: smaller) disk boundaries, so  
the server is back up with the new size.

> Please set your incore log buffer size as a sub-multiple of your log size,
> log size % in core buffer size should be zero (modulo is the operator). This
> is advisable, though not mandatory. Having huge incore buffer size is of no
> help if your on disk log size isn't big. This might solve your problem with
> repair and recovery after some reboots. Make sure that total incore buffer
> size is less than on disk logsize.

Previously, we were using version=1 logs with size=32768b, and then  
mounted using logbufs=8,logbsize=32768.

Now, we are using version=2 logs with size=32768b, and then mounted  
using logbufs=8,logbsize=256k.

If I understand your advise correctly, I should not mount with  
logbsize > 32k, or I should create the filesystem using version=2 logs  
with size=256k. Is this understanding correct?

Are there generic optimization suggestions for I/O-intensive servers  
in general, as far as on-disk and in-memory log buffer sizes are  
concerned?

Please advise.

Thank you very much.

-- 
Federico Sevilla III
F S 3 Consulting Inc.
http://www.fs3.ph

Re: Re: Attempt to Access Beyond End of Device

[willis]

Federico Sevilla III wrote:
"It wasn't an XFS-centric problem, after all"

Adding Federico's comment... Some Adadptec Controller firmware versions will pass incorrect device parameters to the linux kernel. The kernel log output misled me to believe is was a corrupt filesystem or partition map, however it was just an adaptec bug. 

Adaptac has a fix procedure for this at: http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=16914

Below is the my kernel's dmesg output. After following Adaptec's procedure, the errors went away and I was able to mount the filesystem and see all of my existing data.
-----
mount: /dev/sda: can't read superblock
Mount Error at /dev/sdj. Is filesystem realtime? Need a File System? 
attempt to access beyond end of device
sda: rw=0, want=YYYY, limit=XXXX
I/O error in filesystem ("sda") meta-data dev sdj block 0xZZZZ
("xfs_read_buf") error 5 buf count 512
XFS: size check 2 failed
-----

Regards, 

Michael Willis





-- Eric Sandeen wrote : 
Federico Sevilla III wrote:
> Hi,
> 
> We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine with
> two 73.4GB SAS hard drives in hardware RAID 1 with a battery-backed
> cache. We are using the stock Debian 2.6.18-4-686 kernel.

can you send along the results of:

	# xfs_db -r -c "sb 0" -c "print" /dev/sda8?

and
	# cat /proc/partitions

... and does the hardware raid have a funky sector size?

-Eric



--
This message was sent on behalf of willis@arlut.utexas.edu at openSubscriber.com
http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/7550412.html

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Attempt to Access Beyond End of Device

[Eric Sandeen]

On 04/26/2010 03:20 PM, willis@arlut.utexas.edu wrote:
> Federico Sevilla III wrote: "It wasn't an XFS-centric problem, after
> all"
> 
> Adding Federico's comment... Some Adadptec Controller firmware
> versions will pass incorrect device parameters to the linux kernel.
> The kernel log output misled me to believe is was a corrupt
> filesystem or partition map, however it was just an adaptec bug.
> 
> Adaptac has a fix procedure for this at:
> http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=16914

Thanks for the followup; we've seen this a couple times, and it
has always stumped me.

-Eric

>  Below is the my kernel's dmesg output. After following Adaptec's
> procedure, the errors went away and I was able to mount the
> filesystem and see all of my existing data. ----- mount: /dev/sda:
> can't read superblock Mount Error at /dev/sdj. Is filesystem
> realtime? Need a File System? attempt to access beyond end of device 
> sda: rw=0, want=YYYY, limit=XXXX I/O error in filesystem ("sda")
> meta-data dev sdj block 0xZZZZ ("xfs_read_buf") error 5 buf count
> 512 XFS: size check 2 failed -----
> 
> Regards,
> 
> Michael Willis
> 
> 
> 
> 
> 
> -- Eric Sandeen wrote : Federico Sevilla III wrote:
>> Hi,
>> 
>> We've set up Debian GNU/Linux 4.0 "Etch" on an IBM x3400 machine
>> with two 73.4GB SAS hard drives in hardware RAID 1 with a
>> battery-backed cache. We are using the stock Debian 2.6.18-4-686
>> kernel.
> 
> can you send along the results of:
> 
> # xfs_db -r -c "sb 0" -c "print" /dev/sda8?
> 
> and # cat /proc/partitions
> 
> ... and does the hardware raid have a funky sector size?
> 
> -Eric
> 
> 
> 
> -- This message was sent on behalf of willis@arlut.utexas.edu at
> openSubscriber.com 
> http://www.opensubscriber.com/message/linux-xfs@oss.sgi.com/7550412.html
>
>  _______________________________________________ xfs mailing list 
> xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs