If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Ron Leach]

List, may I ask a query about partitions?

Our objective is to run a Debian Wheezy system as a data server using 
an LVM on top of 2 x 3TB discs in RAID-1 configuration.  A first 
attempt had the whole discs used for the data filesystem, using a 
single /dev/md(n), on whole unpartitioned disks.  We've dismantled 
that because of filesystem size problems (it had used only 2TB disks) 
and will make a second attempt and additionally, this time, we want to 
use the array for 3 purposes:

(a) Boot with Grub
(b) Hold the OS
(c) Use the remainder of the disk for the data server, on which we'll 
install an LVM and later grow that with another 2 x 3TB disks.

Assuming I am correct in needing something such as:

/dev/md0 for Grub, (and copied to both physical disks of the RAID-1)
/dev/md1 for the OS, and
/dev/md2 for the data files (on which we'll install the LVM)

then I think we need to partition our disks before creating the array. 
  Is that correct?

The wiki ( https://raid.wiki.kernel.org/index.php/Partition_Types , 
and https://raid.wiki.kernel.org/index.php/RAID_setup#General_setup ) 
is relatively silent on the 'numbers' of partitions that could or 
should be used, if creating a raid on new discs in the circumstances 
where boot 'partitions' and separate OS 'partitions' might be needed. 
  I couldn't see anything in man mdadm to guide me, either, but I 
could have missed something and apologies if I have.

Should I proceed to partition the disks, and then create 3 RAID-1 
arrays (one on each partition-pair), or should I use a different 
technique/layout to hold Grub, the OS, and an expandable LV for the 
datafiles?

I had always assumed that /dev/md(x) always mapped to /dev/sda(y), but 
I have a faint recollection that a discussion on the list a year or so 
ago had suggested that that mapping was not cast in stone, and 
multiple partitions on the physical devices were neither necessary nor 
desirable.  Grateful for any comment,  Ron

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Phil Turmel]

Hi Ron,

On 01/19/2014 01:39 PM, Ron Leach wrote:
> List, may I ask a query about partitions?

Of course!

> Our objective is to run a Debian Wheezy system as a data server using an
> LVM on top of 2 x 3TB discs in RAID-1 configuration.  A first attempt
> had the whole discs used for the data filesystem, using a single
> /dev/md(n), on whole unpartitioned disks.  We've dismantled that because
> of filesystem size problems (it had used only 2TB disks) and will make a
> second attempt and additionally, this time, we want to use the array for
> 3 purposes:
> 
> (a) Boot with Grub
> (b) Hold the OS
> (c) Use the remainder of the disk for the data server, on which we'll
> install an LVM and later grow that with another 2 x 3TB disks.

This kind of setup is fine for light duties (like my personal servers),
but may cause you significant grief if you need to do high-bandwidth
streaming.

(a) is insignificant, as it is only touched at boot or update.

(b) is a random-access workload that tends to spike near and after RAM
exhaustion.  Lots of seeks when busy.

(c) is unspecified here, but streaming workloads often cause RAM
exhaustion as the cache fills.  Then the seeks of the random-access
workload crush the total bandwidth of the drives involved.

> Assuming I am correct in needing something such as:
> 
> /dev/md0 for Grub, (and copied to both physical disks of the RAID-1)
> /dev/md1 for the OS, and

I would use LVM here, too.

> /dev/md2 for the data files (on which we'll install the LVM)
> 
> then I think we need to partition our disks before creating the array.
>  Is that correct?

Yes.

> The wiki ( https://raid.wiki.kernel.org/index.php/Partition_Types , and
> https://raid.wiki.kernel.org/index.php/RAID_setup#General_setup ) is
> relatively silent on the 'numbers' of partitions that could or should be
> used, if creating a raid on new discs in the circumstances where boot
> 'partitions' and separate OS 'partitions' might be needed.  I couldn't
> see anything in man mdadm to guide me, either, but I could have missed
> something and apologies if I have.

Partitioning is not recommended for arrays with heavy-duty workloads.
Mixing workload types on the same spinning disks gets you worst-case
performance for both types.  It doesn't matter for SSDs, but that's
rather expensive in large capacities.

> Should I proceed to partition the disks, and then create 3 RAID-1 arrays
> (one on each partition-pair), or should I use a different
> technique/layout to hold Grub, the OS, and an expandable LV for the
> datafiles?

How were you booting when the two disks were a single array?  Some other
device?  If you can still do that, consider it.

> I had always assumed that /dev/md(x) always mapped to /dev/sda(y), but I
> have a faint recollection that a discussion on the list a year or so ago
> had suggested that that mapping was not cast in stone, and multiple
> partitions on the physical devices were neither necessary nor
> desirable.  Grateful for any comment,  Ron

MD simply makes arrays out of block devices.  It doesn't care,
logically, whether those are whole disks, partitions, loopbacks devices,
or other layered devices.

For the small systems I've built lately, I've set up modest twin SSDs to
handle boot, root, swap, database tablespaces, and mail storage.  Then
added four or more large drives to handle media files.  The SSDs have a
small partition for grub (raid1) and the balance in a single raid1 for
LVM.  The large drives are typically unpartitioned, making a large
raid6, raid10,f3, or raid10,n3.  (In my opinion, large drives aren't
safe with less than double redundancy.)

Oh, and don't forget a backup plan.  Raid redundancy is *not* backup.

If you have bigger goals in mind, ignore me--do whatever Stan says
(seriously).

HTH,

Phil

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[David Brown]

On 19/01/14 21:46, Phil Turmel wrote:
> Hi Ron,
>
> On 01/19/2014 01:39 PM, Ron Leach wrote:
>> List, may I ask a query about partitions?
>
> Of course!
>
>> Our objective is to run a Debian Wheezy system as a data server using an
>> LVM on top of 2 x 3TB discs in RAID-1 configuration.  A first attempt
>> had the whole discs used for the data filesystem, using a single
>> /dev/md(n), on whole unpartitioned disks.  We've dismantled that because
>> of filesystem size problems (it had used only 2TB disks) and will make a
>> second attempt and additionally, this time, we want to use the array for
>> 3 purposes:
>>
>> (a) Boot with Grub
>> (b) Hold the OS
>> (c) Use the remainder of the disk for the data server, on which we'll
>> install an LVM and later grow that with another 2 x 3TB disks.
>
> This kind of setup is fine for light duties (like my personal servers),
> but may cause you significant grief if you need to do high-bandwidth
> streaming.
>
> (a) is insignificant, as it is only touched at boot or update.
>
> (b) is a random-access workload that tends to spike near and after RAM
> exhaustion.  Lots of seeks when busy.
>
> (c) is unspecified here, but streaming workloads often cause RAM
> exhaustion as the cache fills.  Then the seeks of the random-access
> workload crush the total bandwidth of the drives involved.
>
>> Assuming I am correct in needing something such as:
>>
>> /dev/md0 for Grub, (and copied to both physical disks of the RAID-1)
>> /dev/md1 for the OS, and
>
> I would use LVM here, too.
>
>> /dev/md2 for the data files (on which we'll install the LVM)
>>
>> then I think we need to partition our disks before creating the array.
>>   Is that correct?
>
> Yes.
>
>> The wiki ( https://raid.wiki.kernel.org/index.php/Partition_Types , and
>> https://raid.wiki.kernel.org/index.php/RAID_setup#General_setup ) is
>> relatively silent on the 'numbers' of partitions that could or should be
>> used, if creating a raid on new discs in the circumstances where boot
>> 'partitions' and separate OS 'partitions' might be needed.  I couldn't
>> see anything in man mdadm to guide me, either, but I could have missed
>> something and apologies if I have.
>
> Partitioning is not recommended for arrays with heavy-duty workloads.
> Mixing workload types on the same spinning disks gets you worst-case
> performance for both types.  It doesn't matter for SSDs, but that's
> rather expensive in large capacities.
>
>> Should I proceed to partition the disks, and then create 3 RAID-1 arrays
>> (one on each partition-pair), or should I use a different
>> technique/layout to hold Grub, the OS, and an expandable LV for the
>> datafiles?
>
> How were you booting when the two disks were a single array?  Some other
> device?  If you can still do that, consider it.
>
>> I had always assumed that /dev/md(x) always mapped to /dev/sda(y), but I
>> have a faint recollection that a discussion on the list a year or so ago
>> had suggested that that mapping was not cast in stone, and multiple
>> partitions on the physical devices were neither necessary nor
>> desirable.  Grateful for any comment,  Ron
>
> MD simply makes arrays out of block devices.  It doesn't care,
> logically, whether those are whole disks, partitions, loopbacks devices,
> or other layered devices.
>
> For the small systems I've built lately, I've set up modest twin SSDs to
> handle boot, root, swap, database tablespaces, and mail storage.  Then
> added four or more large drives to handle media files.  The SSDs have a
> small partition for grub (raid1) and the balance in a single raid1 for
> LVM.  The large drives are typically unpartitioned, making a large
> raid6, raid10,f3, or raid10,n3.  (In my opinion, large drives aren't
> safe with less than double redundancy.)
>
> Oh, and don't forget a backup plan.  Raid redundancy is *not* backup.
>
> If you have bigger goals in mind, ignore me--do whatever Stan says
> (seriously).
>
> HTH,
>
> Phil

You've made a lot of good points here.  I'd just like to add a couple.

For some types of server, access to the OS partition is pretty minimal - 
all the programs that need to be run are running, and all the files that 
need to be accessed are accessed and cached in ram.  Access to the disk 
is so small that it makes no difference to the main data access (unless 
you are running something with especially tight latency requirements - 
you /could/ be unlucky on occasion).  I once had a firewall/router 
machine that was still running fine a week after the disk controller had 
failed - until some person "helpfully" reset it.

Make sure the server has plenty of ram - you want ram exhaustion and 
swap to be a very rare exception.

When you have plenty of ram, you can reduce access to the OS partition 
by putting things like /tmp, /var/tmp, /var/log, /var/run, etc., on 
tmpfs mounts.  Of course, the files won't survive a crash or restart, 
but it's easier on the disk and faster to access the files.  Make your 
choice according to needs.

And remember, raid is not a backup plan.  (This can't be stressed too 
often!)

David

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Ron Leach]

On 19/01/2014 20:46, Phil Turmel wrote:
> On 01/19/2014 01:39 PM, Ron Leach wrote:
>> Assuming I am correct in needing something such as:
>>
>> /dev/md0 for Grub, (and copied to both physical disks of the RAID-1)
>> /dev/md1 for the OS, and
>
> I would use LVM here, too.
>

I wondered.  Would also enable me to reserve much less space for the 
OS and one or two small services that might be run there.  I'll read 
up on Grub over LVM, which is the only thing I'm not familiar with.

>> Should I proceed to partition the disks, and then create 3 RAID-1 arrays
>> (one on each partition-pair), or should I use a different
>> technique/layout to hold Grub, the OS, and an expandable LV for the
>> datafiles?
>
> How were you booting when the two disks were a single array?  Some other
> device?  If you can still do that, consider it.
>

It was another device - Debian on a separate single 250 GB sda.  With 
only 4 ports on the machine, and a wish to add another 2 x 3TB drives 
for the LVM, it won't be available.  Hence, I was working toward 
putting the OS on the RAID1 array, but now looking into putting the OS 
on the LVM, (as you suggest).

>
> For the small systems I've built lately, I've set up modest twin SSDs to
> handle boot, root, swap, database tablespaces, and mail storage.  Then
> added four or more large drives to handle media files.  The SSDs have a
> small partition for grub (raid1) and the balance in a single raid1 for
> LVM.

Noted, thanks.

>
> Oh, and don't forget a backup plan.  Raid redundancy is *not* backup.
>

Understood.  Incidentally, this server contains project file backups, 
and is primarily a backup repository so that we can recover from 
accidental deletion, or even retrieve previous project information for 
re-examination (should it become necessary).  We are not expecting to 
stream anything (well, not to date, anyway) so I don't expect 
bandwidth problems; we do have a lot of historic data including 
photographs, and project emails.  Relatively low traffic, but large 
capacity, and available on demand.  Additionally, we have a 
full-retention policy, which is imposing the capacity demands on this 
backup server.  We use incremental backup, with rdiff-backup which 
means that files once backed up don't get backed up again unless they 
change.

 From this machine, copies of the backup content will be made, in 
rotation, and held offsite.


> If you have bigger goals in mind, ignore me--do whatever Stan says
> (seriously).

Stan posts on several of the lists we follow, and I always read his 
contributions; many insights over the months and years.

Thanks for the guidance, and for confirming that separate RAID1s does 
imply separate partitions on the physical devices.  I've noted your 
(and David's, in another post) concerns about bandwidth and RAM.  I'll 
look into booting into the LVM, that's new to me.

regards, Ron

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Phil Turmel]

On 01/20/2014 11:53 AM, Ron Leach wrote:
> On 19/01/2014 20:46, Phil Turmel wrote:
>> On 01/19/2014 01:39 PM, Ron Leach wrote:
>>> Assuming I am correct in needing something such as:
>>>
>>> /dev/md0 for Grub, (and copied to both physical disks of the RAID-1)
>>> /dev/md1 for the OS, and
>>
>> I would use LVM here, too.
> 
> I wondered.  Would also enable me to reserve much less space for the OS
> and one or two small services that might be run there.  I'll read up on
> Grub over LVM, which is the only thing I'm not familiar with.

Let me clarify:  /dev/md0 should be a v0.90 or v1.0 raid1 for /boot, no LVM.

/dev/md1 is an LVM PV for the OS.

> Thanks for the guidance, and for confirming that separate RAID1s does
> imply separate partitions on the physical devices.  I've noted your (and
> David's, in another post) concerns about bandwidth and RAM.  I'll look
> into booting into the LVM, that's new to me.

The key is to have both MD and LVM support in your initramfs that gets
loaded by grub with your kernel.  All the major distributions will do
this for you if you create the OS on LVM in the first place.

They'll usually let grub find your root filesystem UUID and put it into
the kernel command line.  You can override that with an explicit
root=LABEL= or root=UUID= setting in /etc/default/grub.

Having root in an LVM allows strange things like relocating and/or
resizing it on the fly.  Particularly handy if physical access is difficult.

HTH,

Phil

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Ron Leach]

On 20/01/2014 18:29, Phil Turmel wrote:
>
> Let me clarify:  /dev/md0 should be a v0.90 or v1.0 raid1 for /boot, no LVM.
 >
 > /dev/md1 is an LVM PV for the OS.


Ok.  That means I need physical partitions for those, on each of the 
drives.

Since I am using 3TB drives, I'll need to use a GPT which, if I 
understand correctly, requires to reveal a dummy 'partition' to make 
the BIOS think there is a boot sector thingy there.

So I'm assuming to pre-partition the physical devices this way:

/dev/sd[x]1 - 64k (? I'm checking the docs) empty, for BIOS to see
/dev/sd[x]2 - 128 MB /boot v0.90/v1.0 RAID1, /dev/md0
/dev/sd[x]3 - remainder ~3 TB for LVM, v1.2 RAID1, /dev/md1

and then make the two RAID1s, and then create the LVM.

>
>
> Having root in an LVM allows strange things like relocating and/or
> resizing it on the fly.  Particularly handy if physical access is difficult.
>

Yes, it was that potential aspect that set me deciding to try your 
approach.  Functional demands on our systems keep changing.

Much appreciated, Ron

Re: If separate md for /boot, OS, and /srv, must 'create' on disks with 3 partns?

[Wilson Jonathan]

On Mon, 2014-01-20 at 19:07 +0000, Ron Leach wrote:
> On 20/01/2014 18:29, Phil Turmel wrote:
> >
> > Let me clarify:  /dev/md0 should be a v0.90 or v1.0 raid1 for /boot, no LVM.
>  >
>  > /dev/md1 is an LVM PV for the OS.
> 
> 
> Ok.  That means I need physical partitions for those, on each of the 
> drives.
> 
> Since I am using 3TB drives, I'll need to use a GPT which, if I 
> understand correctly, requires to reveal a dummy 'partition' to make 
> the BIOS think there is a boot sector thingy there.
> 
> So I'm assuming to pre-partition the physical devices this way:
> 
> /dev/sd[x]1 - 64k (? I'm checking the docs) empty, for BIOS to see

I've set mine to 1MB (bios boot, EF02 in gdisk) this means it aligns
nicely on the 1 MB partition as does its start position 2048 sectors on
a 512e drive.

> /dev/sd[x]2 - 128 MB /boot v0.90/v1.0 RAID1, /dev/md0

I'm not sure here, but I have no problems using a 1.2 metadata
partition... I think the reason it works is that the stuff in biosboot
contains everything needed to boot, the "boot" directory is not needed
with biosboot as its only a temp staging area used to build the boot
loader which is then copied to the biosboot... 

I would check this for yourself just to be sure; I recall I kept it as a
seperate partition due to a miss-understanding and/or it seemed the
"right way" at the time.

> /dev/sd[x]3 - remainder ~3 TB for LVM, v1.2 RAID1, /dev/md1
> 
> and then make the two RAID1s, and then create the LVM.
> 
> >
> >
> > Having root in an LVM allows strange things like relocating and/or
> > resizing it on the fly.  Particularly handy if physical access is difficult.
> >
> 
> Yes, it was that potential aspect that set me deciding to try your 
> approach.  Functional demands on our systems keep changing.
> 
> Much appreciated, Ron
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>