XFS and 3.2TB Partition

XFS and 3.2TB Partition

[Chris Hane]

I am trying to create a 3.2TB partition on my Raid 5.  Is there a 
document that could help?

I have a 3ware 9500 controller and 8 *500GB sata drives configured into 
a single RAID 5 array.

I am running linux 2.6.16 with the 3ware drivers compiled into the kernel.

I've tried a couple of different means to create the partition and 
format the file system with xfs without success (or confidence that I 
haven't done something wrong).

1. FDISK

I've tried fdisk on the array to create the partition; but it forces me 
enter the number of cylinders before letting me create the partition.  I 
enter the largest number of cylinders since I'm not sure how to 
calculate the correct cylinder number across an 8 disk RAID 5 array.

I then create the partition starting at 0 (or whatever the default was) 
and ending at 3500GB.

Once the partition is created this way, I can mkfs.xfs; but I'm a little 
hesitant to use this since I input and arbitrary cylinder number.

Thoughts on what to use for the correct cylinder count with fdisk?

2. PARTED

I've tried to use parted without any success.  Here is what I've tried 
and the errors I get.

 > parted
parted> mklabel gpt
parted> mkpart primary 0 3500GB
parted> quit

ok  - the partition now exists.  If I use ext2 everything works ok.

however, when I run

 > mkfs.xfs /dev/sda1

the file system is formated but is truncated to to 2TB.


Any advice/pointers on how to partition and format a 3.2TB raid 5 array 
would be much appreciated.

Thanks,
Chris....

Re: XFS and 3.2TB Partition

[Eric Sandeen]

Chris Hane wrote:
> 2. PARTED
> 
> I've tried to use parted without any success.  Here is what I've tried 
> and the errors I get.
> 
>  > parted
> parted> mklabel gpt
> parted> mkpart primary 0 3500GB
> parted> quit
> 
> ok  - the partition now exists.  If I use ext2 everything works ok.
> 
> however, when I run
> 
>  > mkfs.xfs /dev/sda1
> 
> the file system is formated but is truncated to to 2TB.

I'd think parted should be able to handle large devices...

mkfs.xfs uses the BLKGETSIZE64 and BLKGETSIZE ioctls to determine device 
size; you might either instrument xfsprogs/libxfs/linux.c and re-run, or 
strace your mkfs command looking for those ioctl calls, to see what size 
it's getting back from the kernel.

-Eric

Re: XFS and 3.2TB Partition

[Shailendra Tripathi]

Hi Chris,

>
> I am trying to create a 3.2TB partition on my Raid 5.  Is there a 
> document that could help?

You can get generic information on partitions and how to create them at
http://www.faqs.org/docs/Linux-mini/Partition.html

> I have a 3ware 9500 controller and 8 *500GB sata drives configured 
> into a single RAID 5 array.
> I am running linux 2.6.16 with the 3ware drivers compiled into the 
> kernel.
>
> I've tried a couple of different means to create the partition and 
> format the file system with xfs without success (or confidence that I 
> haven't done something wrong).
>
> 1. FDISK
>
> I've tried fdisk on the array to create the partition; but it forces 
> me enter the number of cylinders before letting me create the 
> partition.  I enter the largest number of cylinders since I'm not sure 
> how to calculate the correct cylinder number across an 8 disk RAID 5 
> array.
>
> I then create the partition starting at 0 (or whatever the default 
> was) and ending at 3500GB.
>
> Once the partition is created this way, I can mkfs.xfs; but I'm a 
> little hesitant to use this since I input and arbitrary cylinder number.
>
> Thoughts on what to use for the correct cylinder count with fdisk?
>
It is not actually arbitrary. There are various options to create a 
partition - can specify the size, sectors or cylinders. fdisk -l 
/dev/<device> can hopefully give you details about the size of the disk, 
cylinders etc. Now, it is upto to you as to how many partiotions you 
want to create and use. 

> 2. PARTED
>
> I've tried to use parted without any success.  Here is what I've tried 
> and the errors I get.
>
> > parted
> parted> mklabel gpt
> parted> mkpart primary 0 3500GB
> parted> quit
>
> ok  - the partition now exists.  If I use ext2 everything works ok.
>
> however, when I run
>
> > mkfs.xfs /dev/sda1
>
> the file system is formated but is truncated to to 2TB.

XFS can limit the FS size this way when AG count is just 2. xfs_info 
<mnt_point> will list the details about the number of AG count. XFS can 
support about 1 TB in one AG. By default, it calculates the agcount 
automatically unless you force them. Your command line option does not 
suggest this.

The 2 TB limit might be coming, perhaps, because parted uses 4 byte 
addresses data structures. This way, it can support maximum of 2^32  
sectors of  512 size ~ 41 bits ~ 2TB. It is my guess, though.
-shailendra

Re: XFS and 3.2TB Partition

[Russell Cattelan]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

On Thu, 2006-08-31 at 12:57 -0400, Chris Hane wrote:
> 
> I have a 3ware 9500 controller and 8 *500GB sata drives configured
> into 
> a single RAID 5 array.
> 
You might want to google around a bit and verify that your controller
supports
lun's over 2TB.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: XFS and 3.2TB Partition

[Rainer Krienke]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

Am Donnerstag, 31. August 2006 18:57 schrieben Sie:
> I am trying to create a 3.2TB partition on my Raid 5.  Is there a
> document that could help?
>
> I have a 3ware 9500 controller and 8 *500GB sata drives configured into
> a single RAID 5 array.
>

I have a Raid with about 5TB and no problems creating an xfs filesystem on it. 
The system is Novell SLES10 with a 2.6.16.21 kernel. 

At first there was a problem with the raid. The firmware of the raid device 
needed an upgrade. Bevore the upgrade I had a maximum of 2TB. 

In dmesg (or /var/log/boot.msg on SLES10) you should see something like this 
message if the device (sdc here) is handled correctly:

<5>sdc : very big device. try to use READ CAPACITY(16).
<5>SCSI device sdc: 10156243968 512-byte hdwr sectors (5199997 MB)
<5>sdc: Write Protect is off
<7>sdc: Mode Sense: cb 00 00 08
<5>SCSI device sdc: drive cache: write back
<5>sdc : very big device. try to use READ CAPACITY(16).
<5>SCSI device sdc: 10156243968 512-byte hdwr sectors (5199997 MB)

Bevore the firmware update there was an error when trying to read the capacity 
via READ CAPACITY(16).  

I created the partitiions using parted. fdisk did not work.

Have a nice day
Rainer
-- 
---------------------------------------------------------------------------
Rainer Krienke, Universitaet Koblenz, Rechenzentrum, Raum A022
Universitaetsstrasse 1, 56070 Koblenz, Tel: +49 261287 -1312, Fax: -1001312
Mail: krienke@uni-koblenz.de, Web: http://www.uni-koblenz.de/~krienke
Get my public PGP key: http://www.uni-koblenz.de/~krienke/mypgp.html
---------------------------------------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: XFS and 3.2TB Partition

[Chris Hane]

Thank for the input.  I appreciate everyones help!

I believe I am going to end up not partitioning the raid array and using 
it directly (as described in an email which I copied below).

When we get our next large storage machine in (we've are going to need a 
couple over the next year to store CD & DVD ISO Images), I'm going to 
experiment some more with the suggestions everyone has given me here.

As an FYI:  I'm using the latest versions of everything (parted 1.7.1, 
kernel 2.6.16, 3ware raid controller brand new)

Thanks for the help,
Chris....

Peter Grandi wrote:
 >>>> On Thu, 31 Aug 2006 12:57:22 -0400, Chris Hane
 >>>> <chrishane@gmail.com> said:
 >
 > chrishane> I am trying to create a 3.2TB partition on my Raid 5.
 > chrishane> Is there a document that could help?
 >
 > The 9500 is fairly recent, so it should not have a lot of 2TB
 > problems. But there are 2TB limits in several places. For example
 > old versions of the Linux kernel don't support more than 2TB per
 > _filesystem_.
 >
 > But I suspect that you are trying to create partitions in the
 > sense of the MS-DOS/MS-Windows partitioning scheme. Check
 > carefully whether that partitioning scheme allos partitions
 > larger than 2TB :-).
 >
 > Anyhow, usually for very large filesystems you don't need
 > partitions at all. Just use '/dev/sda'. Or check the other
 > partitioning schemes supported by Linux, some may have higher
 > limits.
 >
 > chrishane> I have a 3ware 9500 controller and 8 *500GB sata
 > chrishane> drives configured into a single RAID 5 array.
 >
 > Using RAID5 with 8 drives is a great crime. Nothing to do with
 > your partitioning problems, but since you mentioned it...
 > Consider reading carefully <URL:http://WWW.BAARF.com/>
 >

Rainer Krienke wrote:
> Am Donnerstag, 31. August 2006 18:57 schrieben Sie:
>> I am trying to create a 3.2TB partition on my Raid 5.  Is there a
>> document that could help?
>>
>> I have a 3ware 9500 controller and 8 *500GB sata drives configured into
>> a single RAID 5 array.
>>
> 
> I have a Raid with about 5TB and no problems creating an xfs filesystem on it. 
> The system is Novell SLES10 with a 2.6.16.21 kernel. 
> 
> At first there was a problem with the raid. The firmware of the raid device 
> needed an upgrade. Bevore the upgrade I had a maximum of 2TB. 
> 
> In dmesg (or /var/log/boot.msg on SLES10) you should see something like this 
> message if the device (sdc here) is handled correctly:
> 
> <5>sdc : very big device. try to use READ CAPACITY(16).
> <5>SCSI device sdc: 10156243968 512-byte hdwr sectors (5199997 MB)
> <5>sdc: Write Protect is off
> <7>sdc: Mode Sense: cb 00 00 08
> <5>SCSI device sdc: drive cache: write back
> <5>sdc : very big device. try to use READ CAPACITY(16).
> <5>SCSI device sdc: 10156243968 512-byte hdwr sectors (5199997 MB)
> 
> Bevore the firmware update there was an error when trying to read the capacity 
> via READ CAPACITY(16).  
> 
> I created the partitiions using parted. fdisk did not work.
> 
> Have a nice day
> Rainer

Re: XFS and 3.2TB Partition

[Evan Fraser]

I had that problem when I was using an adaptec aic79xx adapter, using a 
new LSI-Logic one fixed the problem for me.  Could it be a limit with 
your controller/controller driver?

Chris Hane wrote:
>
> I am trying to create a 3.2TB partition on my Raid 5.  Is there a 
> document that could help?
>
> I have a 3ware 9500 controller and 8 *500GB sata drives configured 
> into a single RAID 5 array.
>
> I am running linux 2.6.16 with the 3ware drivers compiled into the 
> kernel.
>
> I've tried a couple of different means to create the partition and 
> format the file system with xfs without success (or confidence that I 
> haven't done something wrong).
>
> 1. FDISK
>
> I've tried fdisk on the array to create the partition; but it forces 
> me enter the number of cylinders before letting me create the 
> partition.  I enter the largest number of cylinders since I'm not sure 
> how to calculate the correct cylinder number across an 8 disk RAID 5 
> array.
>
> I then create the partition starting at 0 (or whatever the default 
> was) and ending at 3500GB.
>
> Once the partition is created this way, I can mkfs.xfs; but I'm a 
> little hesitant to use this since I input and arbitrary cylinder number.
>
> Thoughts on what to use for the correct cylinder count with fdisk?
>
> 2. PARTED
>
> I've tried to use parted without any success.  Here is what I've tried 
> and the errors I get.
>
> > parted
> parted> mklabel gpt
> parted> mkpart primary 0 3500GB
> parted> quit
>
> ok  - the partition now exists.  If I use ext2 everything works ok.
>
> however, when I run
>
> > mkfs.xfs /dev/sda1
>
> the file system is formated but is truncated to to 2TB.
>
>
> Any advice/pointers on how to partition and format a 3.2TB raid 5 
> array would be much appreciated.
>
> Thanks,
> Chris....
>
>
>

-- 
evan@dneg.com
Linux Systems Administrator
Double Negative
tel: +44 (0)20 7534 4400 fax: +44 (0)20 7534 4452
77 shaftesbury avenue, w1d 5du, London

Re: XFS and 3.2TB Partition

[Peter Grandi]

>>> On Thu, 31 Aug 2006 12:57:22 -0400, Chris Hane
>>> <chrishane@gmail.com> said:

chrishane> I've tried to use parted without any success.  Here
chrishane> is what I've tried and the errors I get.

chrishane> parted
    parted> mklabel gpt
    parted> mkpart primary 0 3500GB
    parted> quit

Uhm, I had missed the 'mklabel gpt'. Unlike 'fdisk', which only
supports DOS-style partitions with a 2^32 sector limit (2TiB
indeed) the fairly recent GPT partition scheme (disk labeling in
the Linux kernel jargon) should support just about any size
partitions, with a 2^64 sector limit.

It is somewhat worrying that this happens:

chrishane> ok - the partition now exists.  If I use ext2
chrishane> everything works ok.  however, when I run mkfs.xfs
chrishane> /dev/sda1 the file system is formated but is
chrishane> truncated to to 2TB.

Now, you have succeeded in creating a 3.2TiB partition with
'ext2' on '/dev/sda1' with GPT, and with XFS directly on
'/dev/sda'. It is rather strange that it apparently did not
work with XFS on '/dev/sda1' with GPT.

perhaps 'mkfs.xfs' somehow has problems with the GPT scheme,
but that is indeed strange. It would have been interesting to
see what 'grep /dev/sda /proc/partitions' would have returned.

It might have been interesting to follow this suggestion too

  sandeen> mkfs.xfs uses the BLKGETSIZE64 and BLKGETSIZE ioctls
  sandeen> to determine device size; you might either instrument
  sandeen> xfsprogs/libxfs/linux.c and re-run, or strace your
  sandeen> mkfs command looking for those ioctl calls, to see
  sandeen> what size it's getting back from the kernel.

to see why 'mkfs.ext2' seems to get the right size of a GPT
partition but apparently not 'mkfs.xfs'.

BTW, I hope that you have got a 64 bit system and lots of RAM
for that 3.2TiB filesystem...

  http://OSS.SGI.com/archives/linux-xfs/2005-08/msg00045.html

BTW, for those following this thread: what other ways to create
and manage GPT disk labels are available under GNU/Linux other
than 'parted'?