partitionable md partition size caps at 0.4TB

partitionable md partition size caps at 0.4TB

[Jan Wagner]

Hello,

is it possible to create md partitions larger than 0.4TB?

We have >=9TB RAID-0 systems, and I tried to create a partitionable 
two-partition md with mdadm --auto=mdp2 and partition it into for example 
16MB and ~9TB. For partitioning /dev/md_d0 I have tried sfdisk, fdisk, 
cfdisk, parted, ... Regardless of the partitioning tool, the ~9TB 
partition always ends up as 455780.07MB i.e. 0.4TB.

There is no problem to create a large 9TB single partition on a 
non-partitionable /dev/md0. Is the 445097720 blocks (0.4TB) "cropping" a 
bug or a real limitation with mdp partitionable raid?

Further infos below.

root@abidal:~# uname -a
Linux abidal 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux

root@abidal:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.10
DISTRIB_CODENAME=intrepid
DISTRIB_DESCRIPTION="Ubuntu 8.10"

root@abidal:~# cat /proc/partitions|grep md
  254     0 9035047936 md_d0
  254     1      15622 md_d0p1
  254     2  445097720 md_d0p2

root@abidal:~# sfdisk /dev/md_d0
Checking that no-one is using this disk right now ...
OK

Disk /dev/md_d0: 2258761984 cylinders, 2 heads, 4 sectors/track
Old situation:
Units = cylinders of 4096 bytes, blocks of 1024 bytes, counting from 0

    Device Boot Start     End   #cyls    #blocks   Id  System
/dev/md_d0p1          0+   3905    3906-     15622   83  Linux
/dev/md_d0p2       3906  111278335  111274430  445097720   83  Linux
/dev/md_d0p3          0       -       0          0    0  Empty
/dev/md_d0p4          0       -       0          0    0  Empty

root@abidal:~# mdadm -D /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
root@abidal:~# mdadm -D /dev/md_d0
/dev/md_d0:
         Version : 00.90
   Creation Time : Thu Nov 13 11:58:59 2008
      Raid Level : raid0
      Array Size : 9035047936 (8616.49 GiB 9251.89 GB)
    Raid Devices : 12
   Total Devices : 12
Preferred Minor : 0
     Persistence : Superblock is persistent

     Update Time : Thu Nov 13 11:58:59 2008
           State : clean
  Active Devices : 12
Working Devices : 12
  Failed Devices : 0
   Spare Devices : 0

      Chunk Size : 1024K

            UUID : f1945a73:83d2bea9:b96e9208:11a3b9ec (local to host 
abidal)
          Events : 0.1

     Number   Major   Minor   RaidDevice State
        0       8       17        0      active sync   /dev/sdb1
        1       8       33        1      active sync   /dev/sdc1
        2       8       49        2      active sync   /dev/sdd1
        3       8       65        3      active sync   /dev/sde1
        4       8       81        4      active sync   /dev/sdf1
        5       8       97        5      active sync   /dev/sdg1
        6       8      113        6      active sync   /dev/sdh1
        7       8      129        7      active sync   /dev/sdi1
        8       8      145        8      active sync   /dev/sdj1
        9       8      161        9      active sync   /dev/sdk1
       10       8      177       10      active sync   /dev/sdl1
       11       8      193       11      active sync   /dev/sdm1

thanks,
  - Jan

Re: partitionable md partition size caps at 0.4TB

[Michal Soltys]

Jan Wagner wrote:
> Hello,
> 
> is it possible to create md partitions larger than 0.4TB?
> 
> We have >=9TB RAID-0 systems, and I tried to create a partitionable 
> two-partition md with mdadm --auto=mdp2 and partition it into for 
> example 16MB and ~9TB. For partitioning /dev/md_d0 I have tried sfdisk, 
> fdisk, cfdisk, parted, ... Regardless of the partitioning tool, the ~9TB 
> partition always ends up as 455780.07MB i.e. 0.4TB.
> 
> There is no problem to create a large 9TB single partition on a 
> non-partitionable /dev/md0. Is the 445097720 blocks (0.4TB) "cropping" a 
> bug or a real limitation with mdp partitionable raid?
> 

I'm assuming you tried standard MBR layout - you can't go above ~ 2TB 
limit with it - you need either GPT, or use the device directly (as in 
md0 case). I have one 1.5TB partition running happily on one of my 
systems (still within limits of old MBR layout, but under GPT).

The old md superblock format you used has no such limitation (there're 
others though - maximum 28 components, and 2TB/component). It's really 
better to use one of the 1.x superblocks - check out mdadm's -e option.

If you plan to boot up from a GPT partition on a "legacy" system, you 
will need either syslinux/extlinux or patched grub (and possibly others 
- such as grub2). Not your case as far as I can see, but keep that in 
mind just in case.

- do you have any problems with creating a partition with size of let's 
say 1.8TB and within the first 2TB ?

- check if parted have any problems with creating GPT layout using sizes 
you require (and make sure your kernel is compiled with GPT support).



For a reference, one of my systems:

13:32 > mdadm -D /dev/md/d0p2
/dev/md/d0p2:
         Version : 01.00.03
   Creation Time : Sat Jul 28 12:44:34 2007
      Raid Level : raid5
      Array Size : 1438617600 (1371.97 GiB 1473.14 GB)
   Used Dev Size : 976060928 (465.42 GiB 499.74 GB)
    Raid Devices : 4
   Total Devices : 5
Preferred Minor : 0
     Persistence : Superblock is persistent

     Update Time : Thu Nov 13 13:32:41 2008
           State : clean
  Active Devices : 4
Working Devices : 5
  Failed Devices : 0
   Spare Devices : 1

          Layout : left-symmetric
      Chunk Size : 64K

            Name : d0
            UUID : acf9b8b0:95ba6218:bc4f9bff:67a1fe93
          Events : 2814

     Number   Major   Minor   RaidDevice State
        0       8        3        0      active sync   /dev/sda3
        1       8       19        1      active sync   /dev/sdb3
        2       8       35        2      active sync   /dev/sdc3
        4       8       51        3      active sync   /dev/sdd3

        5       8       67        -      spare   /dev/sde3

13:53 > parted /dev/md/d0
GNU Parted 1.8.8
Using /dev/md/d0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: Unknown (unknown)
Disk /dev/md/d0: 1499GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
  1      17.4kB  25.8GB  25.8GB
  2      25.9GB  1499GB  1473GB  xfs

Re: partitionable md partition size caps at 0.4TB

[Jan Wagner]

Hello Michal,

On Thu, 13 Nov 2008, Michal Soltys wrote:
>> is it possible to create md partitions larger than 0.4TB?
>> 
>> We have >=9TB RAID-0 systems, and I tried to create a partitionable 
>> two-partition md with mdadm --auto=mdp2 and partition it into for example 
>> 16MB and ~9TB. For partitioning /dev/md_d0 I have tried sfdisk, fdisk, 
>> cfdisk, parted, ... Regardless of the partitioning tool, the ~9TB partition 
>> always ends up as 455780.07MB i.e. 0.4TB.
>
> I'm assuming you tried standard MBR layout - you can't go above ~ 2TB limit 
> with it - you need either GPT, or use the device directly (as in md0 case). I 
> have one 1.5TB partition running happily on one of my systems (still within 
> limits of old MBR layout, but under GPT).

Aha, with GPT it works perfectly,

   parted /dev/md_d0 --script mklabel gpt
   parted /dev/md_d0 --script mkpart primary1 0 128M
   yes | parted /dev/md_d0 -- mkpart primary2 128M -1
   ... mkfs.* ...
   parted /dev/md_d0 --script print
   Number  Start   End     Size    File system  Name      Flags
    1      17.4kB  128MB   128MB   ext2         primary1
    2      128MB   9252GB  9252GB  xfs          primary2

and the final size of md_d0p2 is a correct 9252GB. The system does not 
boot from RAID, we just use the raid0 for >>4 Gbps data acquisition. Good 
to know about the newer superblock versions (mdadm -e option).

Thanks for your help!

  - Jan