xfs_growfs doesn't grow on 2TB raid

xfs_growfs doesn't grow on 2TB raid

[jack the ripper]

[-- Attachment #1.1: Type: text/plain, Size: 2521 bytes --]


Hey all,





I ran against a problem today. I had a raid5 array with 5x500GB and
I've grown the array to 6x500GB. So far I always used this method to
add a disk to my raid setup:





    * Partition the disk with one fd partition


    * mdadm --add /dev/mddata1 /dev/sdf1 


    * mdadm --grow /dev/mddata1 --raid-devices=x


    * cryptsetup luksOpen /dev/mddata1 data1 


    * xfs_check /dev/mapper/data1 


    * mount the disk


    * xfs_growfs /dev/mapper/data1 




But at the last step, when I do xfs_growfs, nothing happens. I
found out about the -d option and this is what I get (pay attention to
the last line):





		  
RoSe linux # xfs_growfs /dev/mapper/data1 -d


meta-data=/dev/mapper/data1      isize=256    agcount=9, agsize=61047928 blks


         =                       sectsz=512   attr=2


data     =                       bsize=4096   blocks=488383679, imaxpct=25


         =                       sunit=0      swidth=0 blks


naming   =version 2              bsize=4096   ascii-ci=0


log      =internal               bsize=4096   blocks=32768, version=2


         =                       sectsz=512   sunit=0 blks, lazy-count=0


realtime =none                   extsz=4096   blocks=0, rtextents=0


data size unchanged, skipping	




After searching around on the net I found out that I probably hit
"a" 2TB limit but I checked my kernel support and that seems to be ok:





		  RoSe linux # zgrep -i efi /proc/config.gz


CONFIG_EFI=y


CONFIG_FB_EFI=y


CONFIG_EFI_VARS=y


CONFIG_EFI_PARTITION=y	




Then I caught some hints about mbr and gpt, but is this also valid
with a software raid? That's some kind of a virtual drive, isn't it?
What can I do to bread this 2TB limit, or is it something else that
causes the grow to fail?





Proof that there is more size available:





		  


RoSe linux # mdadm -D /dev/mddata1


/dev/mddata1:


        Version : 0.90


  Creation Time : Sat May 30 19:25:26 2009


     Raid Level : raid5


     Array Size : 2441919680 (2328.80 GiB 2500.53 GB)


	





		  RoSe linux # df --si


Filesystem             Size   Used  Avail Use% Mounted on


/dev/mapper/data1      2.1T   2.0T    63G  97% /mnt/data1
_________________________________________________________________
Je hele online leven op één stek met Windows Live
http://www.microsoft.com/belux/nl/windows/windowslive/default.aspx

[-- Attachment #1.2: Type: text/html, Size: 4152 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Christoph Hellwig]

On Wed, Jun 10, 2009 at 12:52:01PM +0200, jack the ripper wrote:
> 		  RoSe linux # zgrep -i efi /proc/config.gz
> 
> 
> CONFIG_EFI=y
> CONFIG_FB_EFI=y
> CONFIG_EFI_VARS=y
> CONFIG_EFI_PARTITION=y	

Do you have CONFIG_LBD set?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs doesn't grow on 2TB raid

[jack the ripper]

[-- Attachment #1.1: Type: text/plain, Size: 997 bytes --]


When I do:

zgrep LBD /proc/config.gz

I get no output, I also can't find anything about LBD in my /usr/src/linux/.config.

This is the kernel I'm running: 
Linux RoSe 2.6.29-gentoo-r5 #3 SMP Sun Jun 7 19:24:08 CEST 2009 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz GenuineIntel GNU/Linux
Does this indicate a problem or is it normal behaviour?

> Date: Wed, 10 Jun 2009 07:09:02 -0400
> From: hch@infradead.org
> To: evil_metalpeet@hotmail.com
> CC: xfs@oss.sgi.com
> Subject: Re: xfs_growfs doesn't grow on 2TB raid
> 
> On Wed, Jun 10, 2009 at 12:52:01PM +0200, jack the ripper wrote:
> > 		  RoSe linux # zgrep -i efi /proc/config.gz
> > 
> > 
> > CONFIG_EFI=y
> > CONFIG_FB_EFI=y
> > CONFIG_EFI_VARS=y
> > CONFIG_EFI_PARTITION=y	
> 
> Do you have CONFIG_LBD set?
> 

_________________________________________________________________
Lentekriebels? Speel samen met je vrienden de spelletjes die Windows Live je aanbiedt!
http://www.messengerbillboard.be/nl/play

[-- Attachment #1.2: Type: text/html, Size: 1310 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Christoph Hellwig]

On Wed, Jun 10, 2009 at 01:22:17PM +0200, jack the ripper wrote:
> 
> When I do:
> 
> zgrep LBD /proc/config.gz
> 
> I get no output, I also can't find anything about LBD in my /usr/src/linux/.config.
> 
> This is the kernel I'm running: 
> Linux RoSe 2.6.29-gentoo-r5 #3 SMP Sun Jun 7 19:24:08 CEST 2009 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz GenuineIntel GNU/Linux
> Does this indicate a problem or is it normal behaviour?

Oh, it's an x86-64 kernel.  In that case you do not need CONFIG_LBD,
sorry for the noise.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Emmanuel Florac]

Le Wed, 10 Jun 2009 12:52:01 +0200
jack the ripper <evil_metalpeet@hotmail.com> écrivait:

> But at the last step, when I do xfs_growfs, nothing happens. I
> found out about the -d option and this is what I get (pay attention to
> the last line):

Do you run a full 64 bits system? If you're running a 64 bits kernel,
you need a 64 bits xfs_growfs.

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs doesn't grow on 2TB raid

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 1408 bytes --]

Sorry for top-posting, only have Outlook on my notebook.

 

Isn’t 2TB the normal partition size limit? So if you want >2TB partitions, you must use GPT partitioning, “parted” can create such partitions but not fdisk. AFAIK there is no option to simply change a disk from normal partitions to GPT without backup/repartition/restore.

 

mfg zmi

-- 

// Michael Monnerie, Ing.BSc    -----       <http://it-management.at> http://it-management.at

// Tel: +43 (660) 415 6531                    .network.your.ideas.

// PGP Key:         "curl -s  <http://zmi.at/zmi.asc> http://zmi.at/zmi.asc | gpg --import"

// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4

// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


RoSe linux # xfs_growfs /dev/mapper/data1 -d 
meta-data=/dev/mapper/data1      isize=256    agcount=9, agsize=61047928 blks 
         =                       sectsz=512   attr=2 
data     =                       bsize=4096   blocks=488383679, imaxpct=25 
         =                       sunit=0      swidth=0 blks 
naming   =version 2              bsize=4096   ascii-ci=0 
log      =internal               bsize=4096   blocks=32768, version=2 
         =                       sectsz=512   sunit=0 blks, lazy-count=0 
realtime =none                   extsz=4096   blocks=0, rtextents=0 
data size unchanged, skipping

 


[-- Attachment #1.2: Type: text/html, Size: 6148 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs doesn't grow on 2TB raid

[jack the ripper]

[-- Attachment #1.1: Type: text/plain, Size: 2178 bytes --]


2TB is indeed the limit for MBR partitions but I thought this had nothing to do with a "virtual" software raid? The disks an sich have indeed an MBR partition table, but each disk is only 500GB.

Can you confirm that I need GPT for my software raid?

From: michael.monnerie@is.it-management.at
To: evil_metalpeet@hotmail.com
CC: xfs@oss.sgi.com
Subject: RE: xfs_growfs doesn't grow on 2TB raid
Date: Wed, 10 Jun 2009 14:16:35 +0200
















Sorry for top-posting, only have Outlook on my notebook.

 

Isn’t 2TB the normal partition size limit? So if you want
>2TB partitions, you must use GPT partitioning, “parted” can create such
partitions but not fdisk. AFAIK there is no option to simply change a disk from
normal partitions to GPT without backup/repartition/restore.

 

mfg zmi



-- 

// Michael Monnerie, Ing.BSc    -----      http://it-management.at

// Tel: +43 (660) 415 6531                   
.network.your.ideas.

// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg
--import"

// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12
09B4

// Keyserver: wwwkeys.eu.pgp.net                  Key-ID:
1C1209B4






 
  
  RoSe linux # xfs_growfs /dev/mapper/data1
  -d 

  meta-data=/dev/mapper/data1      isize=256   
  agcount=9, agsize=61047928 blks 

           =           
             sectsz=512   attr=2 

  data     =             
           bsize=4096   blocks=488383679,
  imaxpct=25 

           =           
             sunit=0      swidth=0
  blks 

  naming   =version 2             
  bsize=4096   ascii-ci=0 

  log      =internal           
     bsize=4096   blocks=32768, version=2 

           =           
             sectsz=512   sunit=0 blks,
  lazy-count=0 

  realtime =none                 
   extsz=4096   blocks=0, rtextents=0 

  data size unchanged, skipping
  
 




 


_________________________________________________________________
Reageer op foto’s van je vrienden en bekijk hun reacties op de jouwe. Gegarandeerd hilariteit! 
http://www.microsoft.com/belux/nl/windows/windowslive/products/photos.aspx

[-- Attachment #1.2: Type: text/html, Size: 6384 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Michael Weissenbacher]

> Can you confirm that I need GPT for my software raid?
You don't need GPT for software RAID unless one of your disks is >2TB. For example I am running a 3TB XFS file system directly on /dev/md0. Have you done a reboot, maybe the new size has not really been applied? Never tried it with LUKS/cryptsetup though, maybe it won't to above 2GB.

# df -h /dev/md0
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              2.8T  1.5T  1.4T  53% /san1

hth Michael

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Eric Sandeen]

jack the ripper wrote:
> Hey all,

...

> Proof that there is more size available:
> 
> 
> RoSe linux # mdadm -D /dev/mddata1
> /dev/mddata1:
>         Version : 0.90
>   Creation Time : Sat May 30 19:25:26 2009
>      Raid Level : raid5
>      Array Size : 2441919680 (2328.80 GiB 2500.53 GB)


Just to be sure, could you also send the contents of /proc/partitions?

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs doesn't grow on 2TB raid

[jack the ripper]

[-- Attachment #1.1: Type: text/plain, Size: 3043 bytes --]


> cat /proc/partitions
major minor  #blocks  name

   8        0  625131864 sda
   8        1      40131 sda1
   8        2     987997 sda2
   8        3   29302560 sda3
   8        4          1 sda4
   8        5      40131 sda5
   8        6     987966 sda6
   8        7   29302528 sda7
   8        8    9775521 sda8
   8        9    9775521 sda9
   8       10    9775521 sda10
   8       11    9775521 sda11
   8       12    9775521 sda12
   8       13    9775521 sda13
   8       14    9775521 sda14
   8       15    9775521 sda15
 259        0    9775521 sda16
 259        1    9775521 sda17
 259        2   58605088 sda18
   8       16  488386584 sdb
   8       17      40131 sdb1
   8       18     987997 sdb2
   8       19   29302560 sdb3
   8       20          1 sdb4
   8       21      40131 sdb5
   8       22     987966 sdb6
   8       23   29302528 sdb7
   8       24    9775521 sdb8
   8       25    9775521 sdb9
   8       26    9775521 sdb10
   8       27    9775521 sdb11
   8       28    9775521 sdb12
   8       29    9775521 sdb13
   8       30    9775521 sdb14
   8       31    9775521 sdb15
 259        3    9775521 sdb16
 259        4    9775521 sdb17
 259        5   58605088 sdb18
   8       32  488386584 sdc
   8       33  488386552 sdc1
   8       48  488386584 sdd
   8       49  488386552 sdd1
   9        1      40064 md1
   9        3   29302464 md3
 253        0   29301436 dm-0
   8       64  488386584 sde
   8       65  488384001 sde1
   8       80  488386584 sdf
   8       81  488386552 sdf1
   8       96  488386584 sdg
   8       97  488384001 sdg1
   8      112  488386584 sdh
   8      113  488384001 sdh1
   9        5 2441919680 md5
 253        1 1953534716 dm-1

And to make things a tad clearer, my mdstat output:

> cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] 
md5 : active raid5 sdd1[0] sdg1[5] sdc1[4] sde1[3] sdh1[2] sdf1[1]
      2441919680 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      
md3 : active raid1 sda3[0] sdb3[1]
      29302464 blocks [2/2] [UU]
      
md1 : active raid1 sda1[0] sdb1[1]
      40064 blocks [2/2] [UU]
      
unused devices: <none>


> Date: Wed, 10 Jun 2009 08:37:33 -0500
> From: sandeen@sandeen.net
> To: evil_metalpeet@hotmail.com
> CC: xfs@oss.sgi.com
> Subject: Re: xfs_growfs doesn't grow on 2TB raid
> 
> jack the ripper wrote:
> > Hey all,
> 
> ...
> 
> > Proof that there is more size available:
> > 
> > 
> > RoSe linux # mdadm -D /dev/mddata1
> > /dev/mddata1:
> >         Version : 0.90
> >   Creation Time : Sat May 30 19:25:26 2009
> >      Raid Level : raid5
> >      Array Size : 2441919680 (2328.80 GiB 2500.53 GB)
> 
> 
> Just to be sure, could you also send the contents of /proc/partitions?
> 
> -Eric

_________________________________________________________________
Lentekriebels? Speel samen met je vrienden de spelletjes die Windows Live je aanbiedt!
http://www.messengerbillboard.be/nl/play

[-- Attachment #1.2: Type: text/html, Size: 3558 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_growfs doesn't grow on 2TB raid

[Eric Sandeen]

jack the ripper wrote:
> *> cat /proc/partitions*
> major minor  #blocks  name
> 

... snip ...

>    9        5 2441919680 md5  <-- md device
>  253        1 1953534716 dm-1 <-- dm-crypt device
> 

it looks like your md device is larger, but your dm-crypt device is no
larger than it was, so it's not unusual that growfs isn't growing ...

At this point I guess it looks more like a dm-crypt/cryptsetup/LUKS
problem (or at least an admin question).

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: xfs_growfs doesn't grow on 2TB raid

[jack the ripper]

[-- Attachment #1.1: Type: text/plain, Size: 2132 bytes --]


I had not done the reboot (I'm not used to that anymore with linux). I did now though and everything is solved:

RoSe v1per # /root/mount.data1
mdadm: /dev/mddata1 has been started with 6 drives.
Enter LUKS passphrase:
key slot 0 unlocked.
Command successful.

RoSe v1per # xfs_growfs /mnt/data1/
meta-data=/dev/mapper/data1      isize=256    agcount=9, agsize=61047928 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=488383679, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 488383679 to 610479663

RoSe v1per # df --si
Filesystem             Size   Used  Avail Use% Mounted on
udev                    11M   349k    11M   4% /dev
shm                    1.1G      0   1.1G   0% /dev/shm
/dev/mapper/data1      2.6T   1.9T   655G  74% /mnt/data1

Thank you all for helping me so quickly and sorry for bothering all of you with the solution being so simple!

> Date: Wed, 10 Jun 2009 15:05:36 +0200
> From: mw@dermichi.com
> To: evil_metalpeet@hotmail.com
> CC: xfs@oss.sgi.com
> Subject: Re: xfs_growfs doesn't grow on 2TB raid
> 
> > Can you confirm that I need GPT for my software raid?
> You don't need GPT for software RAID unless one of your disks is >2TB. For example I am running a 3TB XFS file system directly on /dev/md0. Have you done a reboot, maybe the new size has not really been applied? Never tried it with LUKS/cryptsetup though, maybe it won't to above 2GB.
> 
> # df -h /dev/md0
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md0              2.8T  1.5T  1.4T  53% /san1
> 
> hth Michael

_________________________________________________________________
Je hele online leven op één stek met Windows Live
http://www.microsoft.com/belux/nl/windows/windowslive/default.aspx

[-- Attachment #1.2: Type: text/html, Size: 3788 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs