Bluestore cluster example

Bluestore cluster example

[Mark Nelson]

Hi all,

A couple of folks have asked me how to setup bluestore clusters for 
performance testing.  I personally am using cbt for this, but you should 
be able to use ceph-disk with some other cluster creation method as well.

For CBT, you really don't need to do much.  In the old newstore days, a 
"block" symlink needed to be created in the osd data dir to link to the 
new block device.  CBT did this when the "newstore_block: True" option 
was set in the cluster section of the cbt yaml file.  This isn't really 
needed anymore, as you can now specify the block, db, and wal devices 
directly in your ceph.conf file.  If your partitions are setup properly 
you can create bluestore clusters without having to do anything beyond 
changing the ceph.conf file (with cbt at least).

Here's a very basic example:

[global]
         enable experimental unrecoverable data corrupting features = 
bluestore rocksdb
         osd objectstore = bluestore

[osd.0]
         host = incerta01.front.sepia.ceph.com
         osd data = /tmp/cbt/mnt/osd-device-0-data
         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal


Here we enable the experimental bluestore and rocksdb features, set the 
objectstore to bluestore, and then in the OSD sections manually set the 
osd data, bluestore block, bluestore block db, and bluestore block wal 
paths.  You might be wondering what all of these are for:

osd data <-- very small directory on FS for bootstrapping OSD.
bluestore block <-- where the actual object data lives
bluestore block db path <-- where rocksdb lives
bluestore block wal path <-- where rocksdb writeahead log lives

And that's basically it.  I've uploaded an example partitioning script, 
ceph.conf file, and cbt yaml configuration file based on actual tests 
I'm running to the examples folder that I've actually been using for 
testing here:

https://github.com/ceph/cbt/tree/master/example/bluestore

Thanks,
Mark

RE: Bluestore cluster example

[Somnath Roy]

Thanks Mark !

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson
Sent: Friday, April 15, 2016 5:06 AM
To: cbt
Cc: ceph-devel
Subject: Bluestore cluster example

Hi all,

A couple of folks have asked me how to setup bluestore clusters for performance testing.  I personally am using cbt for this, but you should be able to use ceph-disk with some other cluster creation method as well.

For CBT, you really don't need to do much.  In the old newstore days, a "block" symlink needed to be created in the osd data dir to link to the new block device.  CBT did this when the "newstore_block: True" option was set in the cluster section of the cbt yaml file.  This isn't really needed anymore, as you can now specify the block, db, and wal devices directly in your ceph.conf file.  If your partitions are setup properly you can create bluestore clusters without having to do anything beyond changing the ceph.conf file (with cbt at least).

Here's a very basic example:

[global]
         enable experimental unrecoverable data corrupting features = bluestore rocksdb
         osd objectstore = bluestore

[osd.0]
         host = incerta01.front.sepia.ceph.com
         osd data = /tmp/cbt/mnt/osd-device-0-data
         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal


Here we enable the experimental bluestore and rocksdb features, set the objectstore to bluestore, and then in the OSD sections manually set the osd data, bluestore block, bluestore block db, and bluestore block wal paths.  You might be wondering what all of these are for:

osd data <-- very small directory on FS for bootstrapping OSD.
bluestore block <-- where the actual object data lives bluestore block db path <-- where rocksdb lives bluestore block wal path <-- where rocksdb writeahead log lives

And that's basically it.  I've uploaded an example partitioning script, ceph.conf file, and cbt yaml configuration file based on actual tests I'm running to the examples folder that I've actually been using for testing here:

https://github.com/ceph/cbt/tree/master/example/bluestore

Thanks,
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: [Cbt] Bluestore cluster example

[Shinobu Kinjo]

Hi Mark,

Just to clarify.
When I build ceph cluster using vstart.sh adding --bluestore option, "osd objectstore" in global section still has filestore.

Anything missing?

Cheers,
Shinobu

----- Original Message -----
From: "Mark Nelson" <mnelson@redhat.com>
To: "cbt" <cbt@ceph.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Sent: Friday, April 15, 2016 9:06:14 PM
Subject: [Cbt] Bluestore cluster example

Hi all,

A couple of folks have asked me how to setup bluestore clusters for 
performance testing.  I personally am using cbt for this, but you should 
be able to use ceph-disk with some other cluster creation method as well.

For CBT, you really don't need to do much.  In the old newstore days, a 
"block" symlink needed to be created in the osd data dir to link to the 
new block device.  CBT did this when the "newstore_block: True" option 
was set in the cluster section of the cbt yaml file.  This isn't really 
needed anymore, as you can now specify the block, db, and wal devices 
directly in your ceph.conf file.  If your partitions are setup properly 
you can create bluestore clusters without having to do anything beyond 
changing the ceph.conf file (with cbt at least).

Here's a very basic example:

[global]
         enable experimental unrecoverable data corrupting features = 
bluestore rocksdb
         osd objectstore = bluestore

[osd.0]
         host = incerta01.front.sepia.ceph.com
         osd data = /tmp/cbt/mnt/osd-device-0-data
         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal


Here we enable the experimental bluestore and rocksdb features, set the 
objectstore to bluestore, and then in the OSD sections manually set the 
osd data, bluestore block, bluestore block db, and bluestore block wal 
paths.  You might be wondering what all of these are for:

osd data <-- very small directory on FS for bootstrapping OSD.
bluestore block <-- where the actual object data lives
bluestore block db path <-- where rocksdb lives
bluestore block wal path <-- where rocksdb writeahead log lives

And that's basically it.  I've uploaded an example partitioning script, 
ceph.conf file, and cbt yaml configuration file based on actual tests 
I'm running to the examples folder that I've actually been using for 
testing here:

https://github.com/ceph/cbt/tree/master/example/bluestore

Thanks,
Mark
_______________________________________________
Cbt mailing list
Cbt@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/cbt-ceph.com

Re: Bluestore cluster example

[Dan van der Ster]

Hi Mark,

On Fri, Apr 15, 2016 at 2:06 PM, Mark Nelson <mnelson@redhat.com> wrote:
> Hi all,
>
> A couple of folks have asked me how to setup bluestore clusters for
> performance testing.  I personally am using cbt for this, but you should be
> able to use ceph-disk with some other cluster creation method as well.
>
> For CBT, you really don't need to do much.  In the old newstore days, a
> "block" symlink needed to be created in the osd data dir to link to the new
> block device.  CBT did this when the "newstore_block: True" option was set
> in the cluster section of the cbt yaml file.  This isn't really needed
> anymore, as you can now specify the block, db, and wal devices directly in
> your ceph.conf file.  If your partitions are setup properly you can create
> bluestore clusters without having to do anything beyond changing the
> ceph.conf file (with cbt at least).
>
> Here's a very basic example:
>
> [global]
>         enable experimental unrecoverable data corrupting features =
> bluestore rocksdb
>         osd objectstore = bluestore
>
> [osd.0]
>         host = incerta01.front.sepia.ceph.com
>         osd data = /tmp/cbt/mnt/osd-device-0-data
>         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
>         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
>         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal

The db and wal paths are optional perf boosting things, right?

If I do ceph-disk prepare --bluestore /dev/sdae, I get:

/dev/sdae :
 /dev/sdae2 ceph block, for /dev/sdae1
 /dev/sdae1 ceph data, active, cluster ceph, osd.413, block /dev/sdae2

-- Dan


>
>
> Here we enable the experimental bluestore and rocksdb features, set the
> objectstore to bluestore, and then in the OSD sections manually set the osd
> data, bluestore block, bluestore block db, and bluestore block wal paths.
> You might be wondering what all of these are for:
>
> osd data <-- very small directory on FS for bootstrapping OSD.
> bluestore block <-- where the actual object data lives
> bluestore block db path <-- where rocksdb lives
> bluestore block wal path <-- where rocksdb writeahead log lives
>
> And that's basically it.  I've uploaded an example partitioning script,
> ceph.conf file, and cbt yaml configuration file based on actual tests I'm
> running to the examples folder that I've actually been using for testing
> here:
>
> https://github.com/ceph/cbt/tree/master/example/bluestore
>
> Thanks,
> Mark
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bluestore cluster example

[Sage Weil]

On Fri, 22 Apr 2016, Dan van der Ster wrote:
> Hi Mark,
> 
> On Fri, Apr 15, 2016 at 2:06 PM, Mark Nelson <mnelson@redhat.com> wrote:
> > Hi all,
> >
> > A couple of folks have asked me how to setup bluestore clusters for
> > performance testing.  I personally am using cbt for this, but you should be
> > able to use ceph-disk with some other cluster creation method as well.
> >
> > For CBT, you really don't need to do much.  In the old newstore days, a
> > "block" symlink needed to be created in the osd data dir to link to the new
> > block device.  CBT did this when the "newstore_block: True" option was set
> > in the cluster section of the cbt yaml file.  This isn't really needed
> > anymore, as you can now specify the block, db, and wal devices directly in
> > your ceph.conf file.  If your partitions are setup properly you can create
> > bluestore clusters without having to do anything beyond changing the
> > ceph.conf file (with cbt at least).
> >
> > Here's a very basic example:
> >
> > [global]
> >         enable experimental unrecoverable data corrupting features =
> > bluestore rocksdb
> >         osd objectstore = bluestore
> >
> > [osd.0]
> >         host = incerta01.front.sepia.ceph.com
> >         osd data = /tmp/cbt/mnt/osd-device-0-data
> >         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
> >         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
> >         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal
> 
> The db and wal paths are optional perf boosting things, right?

Yeah.
 
> If I do ceph-disk prepare --bluestore /dev/sdae, I get:
> 
> /dev/sdae :
>  /dev/sdae2 ceph block, for /dev/sdae1
>  /dev/sdae1 ceph data, active, cluster ceph, osd.413, block /dev/sdae2

Looks right!  ceph-disk isn't smart enough to set up the wal or db devices 
yet.

The wal device should default to something like 128 MB.

The db device would be whatever portion of an SSD you want to allocate to 
storing the bluestore metadata.  My guess is that we'll get further than 
we did with FileStore, so the old 1:4 or 1:5 rule of thumb might be more 
like 1:10, but who knows--this'll require some testing.

sage

Re: Bluestore cluster example

[Dan van der Ster]

On Fri, Apr 22, 2016 at 4:09 PM, Sage Weil <sweil@redhat.com> wrote:
> On Fri, 22 Apr 2016, Dan van der Ster wrote:
>> Hi Mark,
>>
>> On Fri, Apr 15, 2016 at 2:06 PM, Mark Nelson <mnelson@redhat.com> wrote:
>> > Hi all,
>> >
>> > A couple of folks have asked me how to setup bluestore clusters for
>> > performance testing.  I personally am using cbt for this, but you should be
>> > able to use ceph-disk with some other cluster creation method as well.
>> >
>> > For CBT, you really don't need to do much.  In the old newstore days, a
>> > "block" symlink needed to be created in the osd data dir to link to the new
>> > block device.  CBT did this when the "newstore_block: True" option was set
>> > in the cluster section of the cbt yaml file.  This isn't really needed
>> > anymore, as you can now specify the block, db, and wal devices directly in
>> > your ceph.conf file.  If your partitions are setup properly you can create
>> > bluestore clusters without having to do anything beyond changing the
>> > ceph.conf file (with cbt at least).
>> >
>> > Here's a very basic example:
>> >
>> > [global]
>> >         enable experimental unrecoverable data corrupting features =
>> > bluestore rocksdb
>> >         osd objectstore = bluestore
>> >
>> > [osd.0]
>> >         host = incerta01.front.sepia.ceph.com
>> >         osd data = /tmp/cbt/mnt/osd-device-0-data
>> >         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
>> >         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
>> >         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal
>>
>> The db and wal paths are optional perf boosting things, right?
>
> Yeah.
>
>> If I do ceph-disk prepare --bluestore /dev/sdae, I get:
>>
>> /dev/sdae :
>>  /dev/sdae2 ceph block, for /dev/sdae1
>>  /dev/sdae1 ceph data, active, cluster ceph, osd.413, block /dev/sdae2
>
> Looks right!  ceph-disk isn't smart enough to set up the wal or db devices
> yet.
>
> The wal device should default to something like 128 MB.
>
> The db device would be whatever portion of an SSD you want to allocate to
> storing the bluestore metadata.  My guess is that we'll get further than
> we did with FileStore, so the old 1:4 or 1:5 rule of thumb might be more
> like 1:10, but who knows--this'll require some testing.

Cool, I'll start testing wal/db on SSDs with some new machines ~next week.

Just to clarify, if we give no wal, there is no wal, right?
But if we don't give a dedicated db device, where does the db go?

Also, bluestore being experimental in jewel -- I guess that means you
don't promise to support the current on disk format in kraken and
beyond?

Thanks!

Dan




>
> sage
>

Re: Bluestore cluster example

[Sage Weil]

On Fri, 22 Apr 2016, Dan van der Ster wrote:
> On Fri, Apr 22, 2016 at 4:09 PM, Sage Weil <sweil@redhat.com> wrote:
> > On Fri, 22 Apr 2016, Dan van der Ster wrote:
> >> Hi Mark,
> >>
> >> On Fri, Apr 15, 2016 at 2:06 PM, Mark Nelson <mnelson@redhat.com> wrote:
> >> > Hi all,
> >> >
> >> > A couple of folks have asked me how to setup bluestore clusters for
> >> > performance testing.  I personally am using cbt for this, but you should be
> >> > able to use ceph-disk with some other cluster creation method as well.
> >> >
> >> > For CBT, you really don't need to do much.  In the old newstore days, a
> >> > "block" symlink needed to be created in the osd data dir to link to the new
> >> > block device.  CBT did this when the "newstore_block: True" option was set
> >> > in the cluster section of the cbt yaml file.  This isn't really needed
> >> > anymore, as you can now specify the block, db, and wal devices directly in
> >> > your ceph.conf file.  If your partitions are setup properly you can create
> >> > bluestore clusters without having to do anything beyond changing the
> >> > ceph.conf file (with cbt at least).
> >> >
> >> > Here's a very basic example:
> >> >
> >> > [global]
> >> >         enable experimental unrecoverable data corrupting features =
> >> > bluestore rocksdb
> >> >         osd objectstore = bluestore
> >> >
> >> > [osd.0]
> >> >         host = incerta01.front.sepia.ceph.com
> >> >         osd data = /tmp/cbt/mnt/osd-device-0-data
> >> >         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
> >> >         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
> >> >         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal
> >>
> >> The db and wal paths are optional perf boosting things, right?
> >
> > Yeah.
> >
> >> If I do ceph-disk prepare --bluestore /dev/sdae, I get:
> >>
> >> /dev/sdae :
> >>  /dev/sdae2 ceph block, for /dev/sdae1
> >>  /dev/sdae1 ceph data, active, cluster ceph, osd.413, block /dev/sdae2
> >
> > Looks right!  ceph-disk isn't smart enough to set up the wal or db devices
> > yet.
> >
> > The wal device should default to something like 128 MB.
> >
> > The db device would be whatever portion of an SSD you want to allocate to
> > storing the bluestore metadata.  My guess is that we'll get further than
> > we did with FileStore, so the old 1:4 or 1:5 rule of thumb might be more
> > like 1:10, but who knows--this'll require some testing.
> 
> Cool, I'll start testing wal/db on SSDs with some new machines ~next week.
> 
> Just to clarify, if we give no wal, there is no wal, right?
> But if we don't give a dedicated db device, where does the db go?

There are always wal and db "files" from rocksdb, but if there aren't 
separate devices they just end up on the main block device.

> Also, bluestore being experimental in jewel -- I guess that means you
> don't promise to support the current on disk format in kraken and
> beyond?

Correct.

It's *possible* we will include upgrade/conversion code, but it'll sort of 
depend on how much effort it is when it a format change comes along.

The goal is that by kraken it will be stable.

sage

Re: Bluestore cluster example

[Li Tianxiang]

No need to specify wal and db disk path.

But I have run some basic fio tests on v10.1.2, using all SSDs environment.
It seems the results of  NOT desperating those devices (a whole disk) get lower
performance than partitioning the disk and specifying all the 3 disk paths.


 
 
Fio    | ranRd. 4k |randWt.4k | seqRd.64k |seqWt.64k
Parted |   1       |1         |  1        |1   
WholeDisk|0.89     |0.88      |0.98       |0.90





Further tests will be done on 10.2.0 next week.


-------

Outlook web mail is always rejected by ML, annoying :(



-- 
Best Regards!








On 4/22/16, 22:57, "Sage Weil" <ceph-devel-owner@vger.kernel.org on behalf of sweil@redhat.com> wrote:

>On Fri, 22 Apr 2016, Dan van der Ster wrote:
>> On Fri, Apr 22, 2016 at 4:09 PM, Sage Weil <sweil@redhat.com> wrote:
>> > On Fri, 22 Apr 2016, Dan van der Ster wrote:
>> >> Hi Mark,
>> >>
>> >> On Fri, Apr 15, 2016 at 2:06 PM, Mark Nelson <mnelson@redhat.com> wrote:
>> >> > Hi all,
>> >> >
>> >> > A couple of folks have asked me how to setup bluestore clusters for
>> >> > performance testing.  I personally am using cbt for this, but you should be
>> >> > able to use ceph-disk with some other cluster creation method as well.
>> >> >
>> >> > For CBT, you really don't need to do much.  In the old newstore days, a
>> >> > "block" symlink needed to be created in the osd data dir to link to the new
>> >> > block device.  CBT did this when the "newstore_block: True" option was set
>> >> > in the cluster section of the cbt yaml file.  This isn't really needed
>> >> > anymore, as you can now specify the block, db, and wal devices directly in
>> >> > your ceph.conf file.  If your partitions are setup properly you can create
>> >> > bluestore clusters without having to do anything beyond changing the
>> >> > ceph.conf file (with cbt at least).
>> >> >
>> >> > Here's a very basic example:
>> >> >
>> >> > [global]
>> >> >         enable experimental unrecoverable data corrupting features =
>> >> > bluestore rocksdb
>> >> >         osd objectstore = bluestore
>> >> >
>> >> > [osd.0]
>> >> >         host = incerta01.front.sepia.ceph.com
>> >> >         osd data = /tmp/cbt/mnt/osd-device-0-data
>> >> >         bluestore block path = /dev/disk/by-partlabel/osd-device-0-block
>> >> >         bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db
>> >> >         bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal
>> >>
>> >> The db and wal paths are optional perf boosting things, right?
>> >
>> > Yeah.
>> >
>> >> If I do ceph-disk prepare --bluestore /dev/sdae, I get:
>> >>
>> >> /dev/sdae :
>> >>  /dev/sdae2 ceph block, for /dev/sdae1
>> >>  /dev/sdae1 ceph data, active, cluster ceph, osd.413, block /dev/sdae2
>> >
>> > Looks right!  ceph-disk isn't smart enough to set up the wal or db devices
>> > yet.
>> >
>> > The wal device should default to something like 128 MB.
>> >
>> > The db device would be whatever portion of an SSD you want to allocate to
>> > storing the bluestore metadata.  My guess is that we'll get further than
>> > we did with FileStore, so the old 1:4 or 1:5 rule of thumb might be more
>> > like 1:10, but who knows--this'll require some testing.
>> 
>> Cool, I'll start testing wal/db on SSDs with some new machines ~next week.
>> 
>> Just to clarify, if we give no wal, there is no wal, right?
>> But if we don't give a dedicated db device, where does the db go?
>
>There are always wal and db "files" from rocksdb, but if there aren't 
>separate devices they just end up on the main block device.
>
>> Also, bluestore being experimental in jewel -- I guess that means you
>> don't promise to support the current on disk format in kraken and
>> beyond?
>
>Correct.
>
>It's *possible* we will include upgrade/conversion code, but it'll sort of 
>depend on how much effort it is when it a format change comes along.
>
>The goal is that by kraken it will be stable.
>
>sage
>--
>To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html