Multiple disks per server.

Multiple disks per server.

[Mickaël Canévet]

Hi,

I'm testing ceph on 4 old servers.

As there is more then one disk per server available for data (2 with 6 
disks and 2 with 10 disks for a total of 32 disks over 4 nodes), I was 
wondering how to define OSDs.

I have choice between one OSD per disk (32 OSDs on the cluster) or one 
OSD per server with one btrfs filesystem over all disks of the server (4 
OSDs on the cluster). Which one is the best solution ?

In the first case, if I lose one disk, I lose only a small part of 
available space. In the other case, if I lose one disk, I lose the whole 
server (as btrfs filesystem is in stripping) much more space.

On the other hand, if I lose the whole server in the first case, I can 
lose all replicates of a data because they may be on two different OSD 
on the same server.

Is there a way to define OSD groups so that we can be sure that 2 
replicates are not on OSDs of the same group (could be usefull for 
multiple OSDs per server, but also mutliple server per computing room - 
if I lose one whole room, a lot of server, I will be sure that I have 
not lost every replicates).

Thanks a lot.
Mickaël
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multiple disks per server.

[Wido den Hollander]

Hi Mickaël,

I was wondering the same.

When i was reading the Wiki, i found:
http://ceph.newdream.net/wiki/Librados

"Each pool also has a few parameters that define how the object is
stored, namely a replication level (2x, 3x, etc.) and a mapping rule
describing how replicas should be distributed across the storage cluster
(e.g., each replica in a separate rack)."

So there should be a possibility, but i haven't found any configuration
references.

-- 
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: support@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl


On Tue, 2010-05-04 at 12:18 +0000, Mickaël Canévet wrote:
> Hi,
> 
> I'm testing ceph on 4 old servers.
> 
> As there is more then one disk per server available for data (2 with 6 
> disks and 2 with 10 disks for a total of 32 disks over 4 nodes), I was 
> wondering how to define OSDs.
> 
> I have choice between one OSD per disk (32 OSDs on the cluster) or one 
> OSD per server with one btrfs filesystem over all disks of the server (4 
> OSDs on the cluster). Which one is the best solution ?
> 
> In the first case, if I lose one disk, I lose only a small part of 
> available space. In the other case, if I lose one disk, I lose the whole 
> server (as btrfs filesystem is in stripping) much more space.
> 
> On the other hand, if I lose the whole server in the first case, I can 
> lose all replicates of a data because they may be on two different OSD 
> on the same server.
> 
> Is there a way to define OSD groups so that we can be sure that 2 
> replicates are not on OSDs of the same group (could be usefull for 
> multiple OSDs per server, but also mutliple server per computing room - 
> if I lose one whole room, a lot of server, I will be sure that I have 
> not lost every replicates).
> 
> Thanks a lot.
> Mickaël
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multiple disks per server.

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1732 bytes --]

Hi,

On Tue, 4 May 2010, Mickaël Canévet wrote:
> As there is more then one disk per server available for data (2 with 6 disks
> and 2 with 10 disks for a total of 32 disks over 4 nodes), I was wondering how
> to define OSDs.
> 
> I have choice between one OSD per disk (32 OSDs on the cluster) or one OSD per
> server with one btrfs filesystem over all disks of the server (4 OSDs on the
> cluster). Which one is the best solution ?
> 
> In the first case, if I lose one disk, I lose only a small part of available
> space. In the other case, if I lose one disk, I lose the whole server (as
> btrfs filesystem is in stripping) much more space.
> 
> On the other hand, if I lose the whole server in the first case, I can lose
> all replicates of a data because they may be on two different OSD on the same
> server.

Right.  You can make btrfs do some replication (e.g. 2x metadata, 1x data) 
to mitigate the risk somewhat, but it'll use more disk space.  raid[56] 
in btrfs is still a ways off.

> Is there a way to define OSD groups so that we can be sure that 2 replicates
> are not on OSDs of the same group (could be usefull for multiple OSDs per
> server, but also mutliple server per computing room - if I lose one whole
> room, a lot of server, I will be sure that I have not lost every replicates).

This is probably the best route.  The process isn't streamlined, however.  
It involves constructing an appropriate CRUSH rule that includes a 
hierarchy of disks and hosts, and distributes replicas across hosts.  

I threw together a wiki article at

	http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH

Please let me know if you have questions or run into problems.

sage

Re: Multiple disks per server.

[Cláudio Martins]

On Tue, 04 May 2010 14:18:25 +0200 Mickaël Canévet <canevet@embl.fr> wrote:
> Hi,
> 
> I'm testing ceph on 4 old servers.
> 
> As there is more then one disk per server available for data (2 with 6 
> disks and 2 with 10 disks for a total of 32 disks over 4 nodes), I was 
> wondering how to define OSDs.
> 
> I have choice between one OSD per disk (32 OSDs on the cluster) or one 
> OSD per server with one btrfs filesystem over all disks of the server (4 
> OSDs on the cluster). Which one is the best solution ?
> 
> In the first case, if I lose one disk, I lose only a small part of 
> available space. In the other case, if I lose one disk, I lose the whole 
> server (as btrfs filesystem is in stripping) much more space.

 Hi,

 I too am facing a similar dilemma:

 Scenario 1:
 I can set up an MD raid6 array for each OSD box and so can afford up
to 2 simultaneous disk failures without Ceph noticing anything wrong.
When the 3rd drive fails, a long time will be spent redistributing data
across the cluster (though much less time than a simple 25TB raid6
rebuild) . This setup should be quite simple, and a 16 disk raid6
should give generally nice performance, though. I probably would use
2-way data replication (on Ceph config.) for this case.

 Scenario 2:

 I can try to configure 1 OSD per disk. As soon as a drive fails, there
will be data redistribution across the remaining OSDs - but this should
be quite fast, as only the content of a single drive (or slightly more)
has to be redistributed across the cluster (worst case). In this case I
would use 3-way replication for added protection against simultaneous
double drive failures and to compensate for the OSDs not having a raid
array underneath them.

 I can see several potential advantages in "Scenario 2":

 * Greater simplicity and ease of administration, as there's no need to
worry about RAID arrays, their configuration and their possible bugs.
You have one less layer in the stack to worry about, and that has to be
good news.

 * You can replace failed drives with different drives without worrying
about wasted capacity because they are bigger (as you would on raid),
and you can even take advantage of older, smaller drives that would
otherwise go to the trash can. This will give greater liberty when
upgrading hardware, overall.

 * Degradation of available cluster capacity and bandwidth would be
much softer. In fact, assuming that you don't have many power supplies
or mainboards burning up, your cluster will maintain redundancy as
drives go failing. That is, as long as you have more drives than
(amount_of_data * replication_level) your cluster will probably be in a
good, fully redundant state. That should make for better sleep at night.

 * Workloads with small, spread writes should perform better. In a RAID
array those could cause entire stripes to be read, thus requiring data
chunks to be read from a lot of disks just to compute the redundancy
chunks. This one should be quite an advantage for big mail server
workloads, which is one of the workloads I'm interested in.

 * Large write performance should be no worse than with raid, since Ceph
also spreads chunks across OSDs.


 Having said that, there are some aspects about how Ceph would behave
in Scenario 2 that I still have to investigate:

 * If multiple OSDs per node is a well suported option. Do multiple OSDs
per node play well with each other and with a node's resources?

 * If there are issues with network ports/addresses when setting up more
than 1 OSD per node.

 * OSD behaviour when getting I/O errors from its drive -- this is
really the most complex and important one, and the one I wish I could
hear your opinions about:

  Usually, in a RAID array, when there is a fatal failure, the upper
layers will just get permanent I/O errors and you can assume that
storage area is dead and go on with life.
 However, this is frequently not true when you consider single drives
as in Scenario 2, at least for reads: the drive may return read errors
for a small region but still be quite ok for the remaining data.

 So, ideally, a Ceph OSD receiving a read error from the filesystem
would request a copy of the Object in question from another OSD and try
to rewrite it several times before giving up and declaring the drive
dead (1). This is actually what Linux MD does on recent kernels and I
know from my experience that it increases array survivability a lot.
 Background data scrubbing would help a lot with the above, and I guess
BTRFS checksuming will simplify things here.


 Sorry for the huge email, but I hope that what I wrote are valid
points to make Ceph more robust, and hope to know what you think about
them.

Notes:
  (1) Better yet, if the error repeats, it could leave the old backing
file alone, and try to alloc a new one for that object, thus avoiding
declaring the drive completely dead too early.

 Best regards and thanks

Cláudio

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multiple disks per server.

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5462 bytes --]

On Thu, 6 May 2010, Cláudio Martins wrote:
>  I too am facing a similar dilemma:
> 
>  Scenario 1:
>  I can set up an MD raid6 array for each OSD box and so can afford up
> to 2 simultaneous disk failures without Ceph noticing anything wrong.
> When the 3rd drive fails, a long time will be spent redistributing data
> across the cluster (though much less time than a simple 25TB raid6
> rebuild) . This setup should be quite simple, and a 16 disk raid6
> should give generally nice performance, though. I probably would use
> 2-way data replication (on Ceph config.) for this case.
> 
>  Scenario 2:
> 
>  I can try to configure 1 OSD per disk. As soon as a drive fails, there
> will be data redistribution across the remaining OSDs - but this should
> be quite fast, as only the content of a single drive (or slightly more)
> has to be redistributed across the cluster (worst case). In this case I
> would use 3-way replication for added protection against simultaneous
> double drive failures and to compensate for the OSDs not having a raid
> array underneath them.

Disadvantages of Scenario 2:

* If a drive fails, data re-replication bandwidth will be _between_ hosts, 
over the network, instead of being confined to the host or host's RAID 
controller.

* 3x replication will consume more disk than 2x * (N+2)/N RAID overhead.


>  I can see several potential advantages in "Scenario 2":

* I suspect that if you do the math the 3x replication will be more 
reliable.  Notably, losing two hosts (disk aren't the only things that 
fail) can't take out all replicas (even temporarily).  It's a matter of 
tradeoffs...
 
>  * Greater simplicity and ease of administration, as there's no need to
> worry about RAID arrays, their configuration and their possible bugs.
> You have one less layer in the stack to worry about, and that has to be
> good news.
> 
>  * You can replace failed drives with different drives without worrying
> about wasted capacity because they are bigger (as you would on raid),
> and you can even take advantage of older, smaller drives that would
> otherwise go to the trash can. This will give greater liberty when
> upgrading hardware, overall.
> 
>  * Degradation of available cluster capacity and bandwidth would be
> much softer. In fact, assuming that you don't have many power supplies
> or mainboards burning up, your cluster will maintain redundancy as
> drives go failing. That is, as long as you have more drives than
> (amount_of_data * replication_level) your cluster will probably be in a
> good, fully redundant state. That should make for better sleep at night.
> 
>  * Workloads with small, spread writes should perform better. In a RAID
> array those could cause entire stripes to be read, thus requiring data
> chunks to be read from a lot of disks just to compute the redundancy
> chunks. This one should be quite an advantage for big mail server
> workloads, which is one of the workloads I'm interested in.
> 
>  * Large write performance should be no worse than with raid, since Ceph
> also spreads chunks across OSDs.
> 
> 
>  Having said that, there are some aspects about how Ceph would behave
> in Scenario 2 that I still have to investigate:
> 
>  * If multiple OSDs per node is a well suported option. Do multiple OSDs
> per node play well with each other and with a node's resources?

Generally speaking, the cosd daemon is pretty heavily threaded, but all 
threadpools are adjustable in size, so you can tune according to your 
resources.

>  * If there are issues with network ports/addresses when setting up more
> than 1 OSD per node.

None.

>  * OSD behaviour when getting I/O errors from its drive -- this is
> really the most complex and important one, and the one I wish I could
> hear your opinions about:
> 
>   Usually, in a RAID array, when there is a fatal failure, the upper
> layers will just get permanent I/O errors and you can assume that
> storage area is dead and go on with life.
>  However, this is frequently not true when you consider single drives
> as in Scenario 2, at least for reads: the drive may return read errors
> for a small region but still be quite ok for the remaining data.
> 
>  So, ideally, a Ceph OSD receiving a read error from the filesystem
> would request a copy of the Object in question from another OSD and try
> to rewrite it several times before giving up and declaring the drive
> dead (1). This is actually what Linux MD does on recent kernels and I
> know from my experience that it increases array survivability a lot.

There is not yet any specific error handling in the osd.  There is some 
low-level stuff to grab other replicas of an object, but it's only used by 
the scrub function currently when inconsistencies are found.  Some work 
needs to be done to cleanly trigger it on read errors.

In general, btrfs is (will be) pretty good about dealing with individual 
drive errors internally when it has any redundancy.  Currently there's 
only replication (RAID[56] is a work in progress), but moving forward that 
situation will improve.

>  Background data scrubbing would help a lot with the above, and I guess
> BTRFS checksuming will simplify things here.

Yes.. the plan is to leverage the btrfs checksums when comparing replicas 
during the scrub.  Checksums can be verified locally, then compared across 
nodes.

sage