Missing capacity in a RAID 5 array?

Missing capacity in a RAID 5 array?

[Jeff Morrow]

I've created a software RAID 5 array consisting of 8 160 GB drives.  One 
of the drives had prior data I wanted to keep, so I first created a 
7-drive array, copied the data from the original drive to the array, 
then added the 8th drive using raidreconf.  Worked like a charm (took 
about 27 hours, though!)

The OS now reports that the total capacity on /dev/md0 is about 920 GB:

/dev/md0             923029884 172872252 750157632  19% /mnt/store

Now, even if we take into account that the drives are 160 decimal GB, 
which equates to 149 binary GB, my array should still have a total 
capacity somewhere around 7 x 149 = 1043 GB.

Am I really losing 120 GB to reserved space for RAID and/or filesystem 
accounting?  I feel like I'm missing something obvious here.  Can I get 
my full terabyte somehow?

Here's my /etc/raidtab:

raiddev /dev/md0
         raid-level 5
         nr-raid-disks 8
         nr-spare-disks 0
         persistent-superblock 1
         parity-algorithm left-symmetric
         chunk-size 32
         device /dev/hdc
         raid-disk 0
         device /dev/hdd
         raid-disk 1
         device /dev/hde
         raid-disk 2
         device /dev/hdg
         raid-disk 3
         device /dev/hdh
         raid-disk 4
         device /dev/hdi
         raid-disk 5
         device /dev/hdk
         raid-disk 6
         device /dev/hdf
         raid-disk 7

And here's my /proc/mdstat:

Personalities : [raid5]
md0 : active raid5 hdf[7] hdk[6] hdi[5] hdh[4] hdg[3] hde[2] hdd[1] hdc[0]
       1094035712 blocks level 5, 32k chunk, algorithm 2 [8/8] [UUUUUUUU]

unused devices: <none>

Thanks for the help.

Jeff Morrow

Re: Missing capacity in a RAID 5 array?

[Brad Campbell]

Jeff Morrow wrote:
> I've created a software RAID 5 array consisting of 8 160 GB drives.  One 
> of the drives had prior data I wanted to keep, so I first created a 
> 7-drive array, copied the data from the original drive to the array, 
> then added the 8th drive using raidreconf.  Worked like a charm (took 
> about 27 hours, though!)
> 
> The OS now reports that the total capacity on /dev/md0 is about 920 GB:
> 
> /dev/md0             923029884 172872252 750157632  19% /mnt/store
> 
> Now, even if we take into account that the drives are 160 decimal GB, 
> which equates to 149 binary GB, my array should still have a total 
> capacity somewhere around 7 x 149 = 1043 GB.
> 
> Am I really losing 120 GB to reserved space for RAID and/or filesystem 
> accounting?  I feel like I'm missing something obvious here.  Can I get 
> my full terabyte somehow?

What filesystem are you using and how much space does it reserve for Root?
Plus, how much space does the filesystem metadata take up? Perhaps reduce the number of inodes to 
reclaim some space (Can't do that once there is data on there though)


On my 2.1TB I told mke2fs not to reserve any space for root as I was going to lose about 105MB.

Regards,
Brad

Re: Missing capacity in a RAID 5 array?

[Mikael Abrahamsson]

On Wed, 5 May 2004, Jeff Morrow wrote:

> I've created a software RAID 5 array consisting of 8 160 GB drives.  One 
> of the drives had prior data I wanted to keep, so I first created a 
> 7-drive array, copied the data from the original drive to the array, 
> then added the 8th drive using raidreconf.  Worked like a charm (took 
> about 27 hours, though!)
> 
> The OS now reports that the total capacity on /dev/md0 is about 920 GB:
> 
> /dev/md0             923029884 172872252 750157632  19% /mnt/store

Did you grow the filesystem after you grew the raid device? If you used 
ext2s you should look at resize2fs.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Missing capacity in a RAID 5 array?

[Juri Haberland]

Jeff Morrow <jmorrow@jmorrow.org> wrote:
> I've created a software RAID 5 array consisting of 8 160 GB drives.  One 
> of the drives had prior data I wanted to keep, so I first created a 
> 7-drive array, copied the data from the original drive to the array, 
> then added the 8th drive using raidreconf.  Worked like a charm (took 
> about 27 hours, though!)
> 
> The OS now reports that the total capacity on /dev/md0 is about 920 GB:
> 
> /dev/md0             923029884 172872252 750157632  19% /mnt/store
> 
> Now, even if we take into account that the drives are 160 decimal GB, 
> which equates to 149 binary GB, my array should still have a total 
> capacity somewhere around 7 x 149 = 1043 GB.
> 
> Am I really losing 120 GB to reserved space for RAID and/or filesystem 
> accounting?  I feel like I'm missing something obvious here.  Can I get 
> my full terabyte somehow?

So you created a RAID5 array out of 7 disk. The capacity of a RAID5
array of equally sized disk can be calculated like this:
total_size = (n -1) * single_disk_size
which calculates to a total size of 894MB - 960MB (depending on how you
calculate a MegaByte).
The extra disk that you added is most certainly now seen as a spare disk
which adds now capacity to the array, just a instant replacement in case
of failures.

What you should have done:
Create a degraded RAID5 array out of 8 drives with the 8th one marked as
missing/failed. Then copy all the data from the 8th drive to the array.
After this hot-add the 8th drive to degraded array and let it rebuild.

Regards,
Juri

-- 
Juri Haberland  <juri@koschikode.com> 

Re: Missing capacity in a RAID 5 array?

[Derek Vadala]

Brad Campbell said:
> On my 2.1TB I told mke2fs not to reserve any space for root as I was
> going to lose about 105MB [GB?].

As you fill up this filesystem, you're going to see decreased performance,
as the reserved space isn't just "insurance" for root processes, it's also
essential for managing fragmentation. This doesn't apply, as much, if you
are just adding new files, without ever modifying or deleting the old
ones, or if you are working with a static file set altogether.

Re: Missing capacity in a RAID 5 array?

[Jeff Morrow]

> Did you grow the filesystem after you grew the raid device? If you used 
> ext2s you should look at resize2fs.

Yep, that did it.  Many thanks for the quick responses!

Jeff