Re: Adding a proprietary key value store to CEPH

Re: Adding a proprietary key value store to CEPH

[Matt W. Benjamin]

Hi,

We implemented an objectstore factory which does this, and could probably be upstreamed.

Matt

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> Yes, we will take care of both, thanks..
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net] 
> Sent: Wednesday, February 25, 2015 3:25 PM
> To: Somnath Roy
> Cc: Varada Kari; Ceph Development
> Subject: RE: Adding a proprietary key value store to CEPH
> 
> On Wed, 25 Feb 2015, Somnath Roy wrote:
> > Sage,
> > Yes, agreed..
> > But, since k/v store is already added and we are planning to derive
> 
> > from that we didn't change that. Do you want us to implement entire
> 
> > k/vstore to be a dynamically loadable along with the k/vdb
> underneath ?
> 
> Both would be nice.  I confess I haven't looked closely at the EC
> backend implementation, but I expect that this is quite simple: it's
> just a matter of exposing a function that instantiates an instance of
> the ObjectStore or KeyValueDB class, right?
> 
> sage
> 
> 
> > 
> > Thanks & Regards
> > Somnath
> > 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org 
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Wednesday, February 25, 2015 6:46 AM
> > To: Varada Kari
> > Cc: Ceph Development
> > Subject: RE: Adding a proprietary key value store to CEPH
> > 
> > On Wed, 25 Feb 2015, Varada Kari wrote:
> > > Hi Sage,
> > > 
> > > Thanks. Will wait for your reply.
> > > Meanwhile, Can I submit a blue print for this for next release?
> > 
> > Yeah, definitely.
> > 
> > FWIW the plugin layer I think will be most interesting going forward
> is the ObjectStore level, not just the KeyValueDB level...
> > 
> > sage
> > 
> > 
> > > 
> > > Thanks,
> > > Varada
> > > 
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@newdream.net]
> > > Sent: Tuesday, February 24, 2015 10:21 PM
> > > To: Varada Kari
> > > Cc: Ceph Development
> > > Subject: Re: Adding a proprietary key value store to CEPH
> > > 
> > > Hi Varada,
> > > 
> > > On Tue, 24 Feb 2015, Varada Kari wrote:
> > > > Hi Sage,
> > > >
> > > > We are trying to integrate a new proprietary key value store to
> CEPH. To integrate this KV-store, which is a closed source shared
> library, we propose a new class to CEPH called PropDBStore which does
> a dlopen and imports the required symbols. This framework will help in
> integrating vendor specific extensions to CEPH.
> > > >
> > > > The gist of the implementation is as follows.
> > > >
> > > > 1. Implement a wrapper around the proprietary KVStore. Let us
> call it as KVExtension. This is a shared library which implements all
> interfaces required by CEPH KeyValueStore.
> > > > 2. A new class is derived from KeyValueDB called PropDBStore,
> which honors the semantics of KeyvalueStore and KeyValueDB. This class
> acts as mediator between CEPH and KVExtension.  This class transforms
> bufferlist etc... to const char pointers or strings for the extension
> to understand.
> > > > 3. PropDBStore, loads (dlopen) the KVExtension during OSD
> initialization.  Path to the KVExtension can be mentioned in
> ceph.conf.
> > > > 4. Interfaces that needs to be implemented in KVExtension, which
> are imported by the PropDBStore are added in a new header called
> PropDBWrapper.h.  This header contains the signatures for the
> necessary interfaces like init(), close(), submit_transaction(), get()
> and get_iterator(). Similarly for Iterator functionality,
> PropDBIterator.h, which specifies the signatures of seek_to_first (),
> seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore
> includes these headers to import the symbols, using dlsym().
> > > > 5. Choosing the proprietary DB as Backend to the OSD is
> controlled/managed by config options of the ceph (/etc/ceph/ceph.conf)
> like rocksdb or leveldb.
> > > > 6. Rest of the existing functionality is not disturbed by this
> change. Changing the osd backend option will change backend
> implementation. But this change is not dynamic. The type of the
> backend should be chosen at osd creation time and osd will continue
> use that backend till that osd is reformatted again.
> > > > 7. The new KVStore we are trying to integrate works on a raw
> partition, so we divided the osd drive into two partitions. One
> partition is given to osd Meta data (super block, fsid etc...), and
> the other is given to the new db to manage it. OSD partition is now
> not the entire disk, but 2-4GB which needed for the metadata.
> > > 
> > > This sounds like a good approach to link to pretty much any
> external backend (modulo the dlopen part).  I need to check with
> lawyer types before I can promise we'll be able to merge something
> like this into the main ceph.git, though!
> > > 
> > > Thanks-
> > > sage
> > > 
> > > ________________________________
> > > 
> > > PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this message
> in error and that any review, dissemination, distribution, or copying
> of this message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" 
> > > in the body of a message to majordomo@vger.kernel.org More
> majordomo 
> > > info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> 
> > info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

RE: Adding a proprietary key value store to CEPH

[Varada Kari]

Hi Matt,

Can you please upstream the implementation you are mentioning?

Varada

-----Original Message-----
From: Matt W. Benjamin [mailto:matt@cohortfs.com]
Sent: Thursday, February 26, 2015 5:26 AM
To: Somnath Roy
Cc: Varada Kari; Ceph Development; Sage Weil
Subject: Re: Adding a proprietary key value store to CEPH

Hi,

We implemented an objectstore factory which does this, and could probably be upstreamed.

Matt

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> Yes, we will take care of both, thanks..
>
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Wednesday, February 25, 2015 3:25 PM
> To: Somnath Roy
> Cc: Varada Kari; Ceph Development
> Subject: RE: Adding a proprietary key value store to CEPH
>
> On Wed, 25 Feb 2015, Somnath Roy wrote:
> > Sage,
> > Yes, agreed..
> > But, since k/v store is already added and we are planning to derive
>
> > from that we didn't change that. Do you want us to implement entire
>
> > k/vstore to be a dynamically loadable along with the k/vdb
> underneath ?
>
> Both would be nice.  I confess I haven't looked closely at the EC
> backend implementation, but I expect that this is quite simple: it's
> just a matter of exposing a function that instantiates an instance of
> the ObjectStore or KeyValueDB class, right?
>
> sage
>
>
> >
> > Thanks & Regards
> > Somnath
> >
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org
> > [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Wednesday, February 25, 2015 6:46 AM
> > To: Varada Kari
> > Cc: Ceph Development
> > Subject: RE: Adding a proprietary key value store to CEPH
> >
> > On Wed, 25 Feb 2015, Varada Kari wrote:
> > > Hi Sage,
> > >
> > > Thanks. Will wait for your reply.
> > > Meanwhile, Can I submit a blue print for this for next release?
> >
> > Yeah, definitely.
> >
> > FWIW the plugin layer I think will be most interesting going forward
> is the ObjectStore level, not just the KeyValueDB level...
> >
> > sage
> >
> >
> > >
> > > Thanks,
> > > Varada
> > >
> > > -----Original Message-----
> > > From: Sage Weil [mailto:sage@newdream.net]
> > > Sent: Tuesday, February 24, 2015 10:21 PM
> > > To: Varada Kari
> > > Cc: Ceph Development
> > > Subject: Re: Adding a proprietary key value store to CEPH
> > >
> > > Hi Varada,
> > >
> > > On Tue, 24 Feb 2015, Varada Kari wrote:
> > > > Hi Sage,
> > > >
> > > > We are trying to integrate a new proprietary key value store to
> CEPH. To integrate this KV-store, which is a closed source shared
> library, we propose a new class to CEPH called PropDBStore which does
> a dlopen and imports the required symbols. This framework will help in
> integrating vendor specific extensions to CEPH.
> > > >
> > > > The gist of the implementation is as follows.
> > > >
> > > > 1. Implement a wrapper around the proprietary KVStore. Let us
> call it as KVExtension. This is a shared library which implements all
> interfaces required by CEPH KeyValueStore.
> > > > 2. A new class is derived from KeyValueDB called PropDBStore,
> which honors the semantics of KeyvalueStore and KeyValueDB. This class
> acts as mediator between CEPH and KVExtension.  This class transforms
> bufferlist etc... to const char pointers or strings for the extension
> to understand.
> > > > 3. PropDBStore, loads (dlopen) the KVExtension during OSD
> initialization.  Path to the KVExtension can be mentioned in
> ceph.conf.
> > > > 4. Interfaces that needs to be implemented in KVExtension, which
> are imported by the PropDBStore are added in a new header called
> PropDBWrapper.h.  This header contains the signatures for the
> necessary interfaces like init(), close(), submit_transaction(), get()
> and get_iterator(). Similarly for Iterator functionality,
> PropDBIterator.h, which specifies the signatures of seek_to_first (),
> seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore
> includes these headers to import the symbols, using dlsym().
> > > > 5. Choosing the proprietary DB as Backend to the OSD is
> controlled/managed by config options of the ceph (/etc/ceph/ceph.conf)
> like rocksdb or leveldb.
> > > > 6. Rest of the existing functionality is not disturbed by this
> change. Changing the osd backend option will change backend
> implementation. But this change is not dynamic. The type of the
> backend should be chosen at osd creation time and osd will continue
> use that backend till that osd is reformatted again.
> > > > 7. The new KVStore we are trying to integrate works on a raw
> partition, so we divided the osd drive into two partitions. One
> partition is given to osd Meta data (super block, fsid etc...), and
> the other is given to the new db to manage it. OSD partition is now
> not the entire disk, but 2-4GB which needed for the metadata.
> > >
> > > This sounds like a good approach to link to pretty much any
> external backend (modulo the dlopen part).  I need to check with
> lawyer types before I can promise we'll be able to merge something
> like this into the main ceph.git, though!
> > >
> > > Thanks-
> > > sage
> > >
> > > ________________________________
> > >
> > > PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this message
> in error and that any review, dissemination, distribution, or copying
> of this message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel"
> > > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > >
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
>
> > info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
>
> > info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org More majordomo info
> at  http://vger.kernel.org/majordomo-info.html

--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689
fax.  734-769-8938
cel.  734-216-5309

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Adding a proprietary key value store to CEPH

[Varada Kari]

Hi Sage,

We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.

The gist of the implementation is as follows.

1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.

Please share your thoughts around this.
Thanks,
Varada



________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: Adding a proprietary key value store to CEPH

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3678 bytes --]

Hi,

I'm curious about the reasons why the key/value store you mention is not published as Free Software. Is it because it implements a proprietary interface to a specific hardware ? Because it has additional functionalities comparied to rocksdb etc. ? Because it performs better under some workloads ?

Cheers

On 24/02/2015 14:20, Varada Kari wrote:
> Hi Sage,
> 
> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> 
> The gist of the implementation is as follows.
> 
> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> 
> Please share your thoughts around this.
> Thanks,
> Varada
> 
> 
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: Adding a proprietary key value store to CEPH

[Somnath Roy]

Hi Loic,
This is an effort to make ceph interface pluggable to any proprietary k/v db available. The integrator has to implement a shim layer (dynamically loadable) by implementing these interfaces. That shim layer can do specific job for the k/v db of theirs.
Now, regarding our k/v db, yes, it is written keeping in mind that backend will be flash not HDD. This is the major difference between leveldb/rocksdb etc. Our db reduces the flash WA dramatically and the performance also should be similar or better than rocksdb. 
Also, I think there should more of this proprietary dbs that people want to integrate with Ceph as I don't think leveldb/rocksdb will not be able to serve all kind of workload.

Thanks & Regards
Somnath 

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
Sent: Tuesday, February 24, 2015 6:30 AM
To: Varada Kari; Ceph Development
Subject: Re: Adding a proprietary key value store to CEPH

Hi,

I'm curious about the reasons why the key/value store you mention is not published as Free Software. Is it because it implements a proprietary interface to a specific hardware ? Because it has additional functionalities comparied to rocksdb etc. ? Because it performs better under some workloads ?

Cheers

On 24/02/2015 14:20, Varada Kari wrote:
> Hi Sage,
> 
> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> 
> The gist of the implementation is as follows.
> 
> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> 
> Please share your thoughts around this.
> Thanks,
> Varada
> 
> 
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 

--
Loïc Dachary, Artisan Logiciel Libre

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Adding a proprietary key value store to CEPH

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 5175 bytes --]

Hi,

On 24/02/2015 17:13, Somnath Roy wrote:> Hi Loic,
> This is an effort to make ceph interface pluggable to any proprietary k/v db available. The integrator has to implement a shim layer (dynamically loadable) by implementing these interfaces. That shim layer can do specific job for the k/v db of theirs.
> Now, regarding our k/v db, yes, it is written keeping in mind that backend will be flash not HDD. This is the major difference between leveldb/rocksdb etc. Our db reduces the flash WA dramatically and the performance also should be similar or better than rocksdb. 
> Also, I think there should more of this proprietary dbs that people want to integrate with Ceph as I don't think leveldb/rocksdb will not be able to serve all kind of workload.

Thanks for sharing these details :-) Would this db be specific to a line of product, for instance by making ioctl calls that only a specific driver for a specific hardware would understand ? Or is this a db that is designed to optimize workloads for flash drives using only standard and documented API or system calls ?

> Thanks & Regards
> Somnath 
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
> Sent: Tuesday, February 24, 2015 6:30 AM
> To: Varada Kari; Ceph Development
> Subject: Re: Adding a proprietary key value store to CEPH
> 
> Hi,
> 
> I'm curious about the reasons why the key/value store you mention is not published as Free Software. Is it because it implements a proprietary interface to a specific hardware ? Because it has additional functionalities comparied to rocksdb etc. ? Because it performs better under some workloads ?
> 
> Cheers
> 
> On 24/02/2015 14:20, Varada Kari wrote:
>> Hi Sage,
>>
>> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
>>
>> The gist of the implementation is as follows.
>>
>> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
>> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
>> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
>> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
>> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
>> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
>> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
>>
>> Please share your thoughts around this.
>> Thanks,
>> Varada
>>
>>
>>
>> ________________________________
>>
>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

RE: Adding a proprietary key value store to CEPH

[Varada Kari]

Hi Loic,

Yes,  db is designed to optimize the workloads on flash backends and uses only standard interfaces and system calls to achieve that. 

Varada

-----Original Message-----
From: Loic Dachary [mailto:loic@dachary.org] 
Sent: Tuesday, February 24, 2015 9:57 PM
To: Somnath Roy; Varada Kari; Ceph Development
Subject: Re: Adding a proprietary key value store to CEPH

Hi,

On 24/02/2015 17:13, Somnath Roy wrote:> Hi Loic,
> This is an effort to make ceph interface pluggable to any proprietary k/v db available. The integrator has to implement a shim layer (dynamically loadable) by implementing these interfaces. That shim layer can do specific job for the k/v db of theirs.
> Now, regarding our k/v db, yes, it is written keeping in mind that backend will be flash not HDD. This is the major difference between leveldb/rocksdb etc. Our db reduces the flash WA dramatically and the performance also should be similar or better than rocksdb. 
> Also, I think there should more of this proprietary dbs that people want to integrate with Ceph as I don't think leveldb/rocksdb will not be able to serve all kind of workload.

Thanks for sharing these details :-) Would this db be specific to a line of product, for instance by making ioctl calls that only a specific driver for a specific hardware would understand ? Or is this a db that is designed to optimize workloads for flash drives using only standard and documented API or system calls ?

> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
> Sent: Tuesday, February 24, 2015 6:30 AM
> To: Varada Kari; Ceph Development
> Subject: Re: Adding a proprietary key value store to CEPH
> 
> Hi,
> 
> I'm curious about the reasons why the key/value store you mention is not published as Free Software. Is it because it implements a proprietary interface to a specific hardware ? Because it has additional functionalities comparied to rocksdb etc. ? Because it performs better under some workloads ?
> 
> Cheers
> 
> On 24/02/2015 14:20, Varada Kari wrote:
>> Hi Sage,
>>
>> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
>>
>> The gist of the implementation is as follows.
>>
>> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
>> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
>> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
>> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
>> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
>> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
>> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
>>
>> Please share your thoughts around this.
>> Thanks,
>> Varada
>>
>>
>>
>> ________________________________
>>
>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 

--
Loïc Dachary, Artisan Logiciel Libre

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Adding a proprietary key value store to CEPH

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 6675 bytes --]



On 24/02/2015 17:50, Varada Kari wrote:
> Hi Loic,
> 
> Yes,  db is designed to optimize the workloads on flash backends and uses only standard interfaces and system calls to achieve that. 

I find the Cinder (https://wiki.openstack.org/wiki/Cinder) approach to proprietary drivers support a good source of inspiration. They had to find the right balance between the needs of the vendors and having a reference implementation that is both Free Software and matching the needs of all users. For instance https://wiki.openstack.org/wiki/Cinder/how-to-contribute-a-driver states:

* You must implement all of the methods that exist as core features http://docs.openstack.org/developer/cinder/devref/drivers.html
* Third party tests are required https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers

I suspect people more involved in OpenStack than I am can share interesting stories ;-)

My 2cts.

> Varada
> 
> -----Original Message-----
> From: Loic Dachary [mailto:loic@dachary.org] 
> Sent: Tuesday, February 24, 2015 9:57 PM
> To: Somnath Roy; Varada Kari; Ceph Development
> Subject: Re: Adding a proprietary key value store to CEPH
> 
> Hi,
> 
> On 24/02/2015 17:13, Somnath Roy wrote:> Hi Loic,
>> This is an effort to make ceph interface pluggable to any proprietary k/v db available. The integrator has to implement a shim layer (dynamically loadable) by implementing these interfaces. That shim layer can do specific job for the k/v db of theirs.
>> Now, regarding our k/v db, yes, it is written keeping in mind that backend will be flash not HDD. This is the major difference between leveldb/rocksdb etc. Our db reduces the flash WA dramatically and the performance also should be similar or better than rocksdb. 
>> Also, I think there should more of this proprietary dbs that people want to integrate with Ceph as I don't think leveldb/rocksdb will not be able to serve all kind of workload.
> 
> Thanks for sharing these details :-) Would this db be specific to a line of product, for instance by making ioctl calls that only a specific driver for a specific hardware would understand ? Or is this a db that is designed to optimize workloads for flash drives using only standard and documented API or system calls ?
> 
>> Thanks & Regards
>> Somnath
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org 
>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Loic Dachary
>> Sent: Tuesday, February 24, 2015 6:30 AM
>> To: Varada Kari; Ceph Development
>> Subject: Re: Adding a proprietary key value store to CEPH
>>
>> Hi,
>>
>> I'm curious about the reasons why the key/value store you mention is not published as Free Software. Is it because it implements a proprietary interface to a specific hardware ? Because it has additional functionalities comparied to rocksdb etc. ? Because it performs better under some workloads ?
>>
>> Cheers
>>
>> On 24/02/2015 14:20, Varada Kari wrote:
>>> Hi Sage,
>>>
>>> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
>>>
>>> The gist of the implementation is as follows.
>>>
>>> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
>>> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
>>> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
>>> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
>>> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
>>> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
>>> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
>>>
>>> Please share your thoughts around this.
>>> Thanks,
>>> Varada
>>>
>>>
>>>
>>> ________________________________
>>>
>>> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
>>> in the body of a message to majordomo@vger.kernel.org More majordomo 
>>> info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
> 
> --
> Loïc Dachary, Artisan Logiciel Libre
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Adding a proprietary key value store to CEPH

[Sage Weil]

Hi Varada,

On Tue, 24 Feb 2015, Varada Kari wrote:
> Hi Sage,
> 
> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> 
> The gist of the implementation is as follows.
> 
> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.

This sounds like a good approach to link to pretty much any external 
backend (modulo the dlopen part).  I need to check with lawyer types 
before I can promise we'll be able to merge something like this into the 
main ceph.git, though!

Thanks-
sage

RE: Adding a proprietary key value store to CEPH

[Varada Kari]

Hi Sage,

Thanks. Will wait for your reply.
Meanwhile, Can I submit a blue print for this for next release?

Thanks,
Varada

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Tuesday, February 24, 2015 10:21 PM
To: Varada Kari
Cc: Ceph Development
Subject: Re: Adding a proprietary key value store to CEPH

Hi Varada,

On Tue, 24 Feb 2015, Varada Kari wrote:
> Hi Sage,
>
> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
>
> The gist of the implementation is as follows.
>
> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.

This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!

Thanks-
sage

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Adding a proprietary key value store to CEPH

[Sage Weil]

On Wed, 25 Feb 2015, Varada Kari wrote:
> Hi Sage,
> 
> Thanks. Will wait for your reply.
> Meanwhile, Can I submit a blue print for this for next release?

Yeah, definitely.

FWIW the plugin layer I think will be most interesting going forward is 
the ObjectStore level, not just the KeyValueDB level...

sage


> 
> Thanks,
> Varada
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Tuesday, February 24, 2015 10:21 PM
> To: Varada Kari
> Cc: Ceph Development
> Subject: Re: Adding a proprietary key value store to CEPH
> 
> Hi Varada,
> 
> On Tue, 24 Feb 2015, Varada Kari wrote:
> > Hi Sage,
> >
> > We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> >
> > The gist of the implementation is as follows.
> >
> > 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> > 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> > 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> > 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> > 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> > 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> > 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> 
> This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!
> 
> Thanks-
> sage
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: Adding a proprietary key value store to CEPH

[Somnath Roy]

Sage,
Yes, agreed..
But, since k/v store is already added and we are planning to derive from that we didn't change that.
Do you want us to implement entire k/vstore to be a dynamically loadable along with the k/vdb underneath ?

Thanks & Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Wednesday, February 25, 2015 6:46 AM
To: Varada Kari
Cc: Ceph Development
Subject: RE: Adding a proprietary key value store to CEPH

On Wed, 25 Feb 2015, Varada Kari wrote:
> Hi Sage,
> 
> Thanks. Will wait for your reply.
> Meanwhile, Can I submit a blue print for this for next release?

Yeah, definitely.

FWIW the plugin layer I think will be most interesting going forward is the ObjectStore level, not just the KeyValueDB level...

sage


> 
> Thanks,
> Varada
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Tuesday, February 24, 2015 10:21 PM
> To: Varada Kari
> Cc: Ceph Development
> Subject: Re: Adding a proprietary key value store to CEPH
> 
> Hi Varada,
> 
> On Tue, 24 Feb 2015, Varada Kari wrote:
> > Hi Sage,
> >
> > We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> >
> > The gist of the implementation is as follows.
> >
> > 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> > 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> > 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> > 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> > 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> > 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> > 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> 
> This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!
> 
> Thanks-
> sage
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Adding a proprietary key value store to CEPH

[Sage Weil]

On Wed, 25 Feb 2015, Somnath Roy wrote:
> Sage,
> Yes, agreed..
> But, since k/v store is already added and we are planning to derive from 
> that we didn't change that. Do you want us to implement entire k/vstore 
> to be a dynamically loadable along with the k/vdb underneath ?

Both would be nice.  I confess I haven't looked closely at the EC 
backend implementation, but I expect that this is quite simple: it's just 
a matter of exposing a function that instantiates an instance of the 
ObjectStore or KeyValueDB class, right?

sage


> 
> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Wednesday, February 25, 2015 6:46 AM
> To: Varada Kari
> Cc: Ceph Development
> Subject: RE: Adding a proprietary key value store to CEPH
> 
> On Wed, 25 Feb 2015, Varada Kari wrote:
> > Hi Sage,
> > 
> > Thanks. Will wait for your reply.
> > Meanwhile, Can I submit a blue print for this for next release?
> 
> Yeah, definitely.
> 
> FWIW the plugin layer I think will be most interesting going forward is the ObjectStore level, not just the KeyValueDB level...
> 
> sage
> 
> 
> > 
> > Thanks,
> > Varada
> > 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Tuesday, February 24, 2015 10:21 PM
> > To: Varada Kari
> > Cc: Ceph Development
> > Subject: Re: Adding a proprietary key value store to CEPH
> > 
> > Hi Varada,
> > 
> > On Tue, 24 Feb 2015, Varada Kari wrote:
> > > Hi Sage,
> > >
> > > We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> > >
> > > The gist of the implementation is as follows.
> > >
> > > 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> > > 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> > > 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> > > 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> > > 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> > > 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> > > 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> > 
> > This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!
> > 
> > Thanks-
> > sage
> > 
> > ________________________________
> > 
> > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: Adding a proprietary key value store to CEPH

[Somnath Roy]

Yes, we will take care of both, thanks..

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net] 
Sent: Wednesday, February 25, 2015 3:25 PM
To: Somnath Roy
Cc: Varada Kari; Ceph Development
Subject: RE: Adding a proprietary key value store to CEPH

On Wed, 25 Feb 2015, Somnath Roy wrote:
> Sage,
> Yes, agreed..
> But, since k/v store is already added and we are planning to derive 
> from that we didn't change that. Do you want us to implement entire 
> k/vstore to be a dynamically loadable along with the k/vdb underneath ?

Both would be nice.  I confess I haven't looked closely at the EC backend implementation, but I expect that this is quite simple: it's just a matter of exposing a function that instantiates an instance of the ObjectStore or KeyValueDB class, right?

sage


> 
> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org 
> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Wednesday, February 25, 2015 6:46 AM
> To: Varada Kari
> Cc: Ceph Development
> Subject: RE: Adding a proprietary key value store to CEPH
> 
> On Wed, 25 Feb 2015, Varada Kari wrote:
> > Hi Sage,
> > 
> > Thanks. Will wait for your reply.
> > Meanwhile, Can I submit a blue print for this for next release?
> 
> Yeah, definitely.
> 
> FWIW the plugin layer I think will be most interesting going forward is the ObjectStore level, not just the KeyValueDB level...
> 
> sage
> 
> 
> > 
> > Thanks,
> > Varada
> > 
> > -----Original Message-----
> > From: Sage Weil [mailto:sage@newdream.net]
> > Sent: Tuesday, February 24, 2015 10:21 PM
> > To: Varada Kari
> > Cc: Ceph Development
> > Subject: Re: Adding a proprietary key value store to CEPH
> > 
> > Hi Varada,
> > 
> > On Tue, 24 Feb 2015, Varada Kari wrote:
> > > Hi Sage,
> > >
> > > We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
> > >
> > > The gist of the implementation is as follows.
> > >
> > > 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> > > 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> > > 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> > > 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> > > 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> > > 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> > > 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.
> > 
> > This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!
> > 
> > Thanks-
> > sage
> > 
> > ________________________________
> > 
> > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: Adding a proprietary key value store to CEPH

[Varada Kari]

Hi Sage,

Any news on legal side to this approach? :-). I am kind of ready with the first cut version of the code, if you say okay, will submit a pull request soon.

Varada

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Tuesday, February 24, 2015 10:21 PM
To: Varada Kari
Cc: Ceph Development
Subject: Re: Adding a proprietary key value store to CEPH

Hi Varada,

On Tue, 24 Feb 2015, Varada Kari wrote:
> Hi Sage,
>
> We are trying to integrate a new proprietary key value store to CEPH. To integrate this KV-store, which is a closed source shared library, we propose a new class to CEPH called PropDBStore which does a dlopen and imports the required symbols. This framework will help in integrating vendor specific extensions to CEPH.
>
> The gist of the implementation is as follows.
>
> 1. Implement a wrapper around the proprietary KVStore. Let us call it as KVExtension. This is a shared library which implements all interfaces required by CEPH KeyValueStore.
> 2. A new class is derived from KeyValueDB called PropDBStore, which honors the semantics of KeyvalueStore and KeyValueDB. This class acts as mediator between CEPH and KVExtension.  This class transforms bufferlist etc... to const char pointers or strings for the extension to understand.
> 3. PropDBStore, loads (dlopen) the KVExtension during OSD initialization.  Path to the KVExtension can be mentioned in ceph.conf.
> 4. Interfaces that needs to be implemented in KVExtension, which are imported by the PropDBStore are added in a new header called PropDBWrapper.h.  This header contains the signatures for the necessary interfaces like init(), close(), submit_transaction(), get() and get_iterator(). Similarly for Iterator functionality, PropDBIterator.h, which specifies the signatures of seek_to_first (), seek_to_last(), lower_bound() and upper_bound() etc...  PropDBStore includes these headers to import the symbols, using dlsym().
> 5. Choosing the proprietary DB as Backend to the OSD is controlled/managed by config options of the ceph (/etc/ceph/ceph.conf) like rocksdb or leveldb.
> 6. Rest of the existing functionality is not disturbed by this change. Changing the osd backend option will change backend implementation. But this change is not dynamic. The type of the backend should be chosen at osd creation time and osd will continue use that backend till that osd is reformatted again.
> 7. The new KVStore we are trying to integrate works on a raw partition, so we divided the osd drive into two partitions. One partition is given to osd Meta data (super block, fsid etc...), and the other is given to the new db to manage it. OSD partition is now not the entire disk, but 2-4GB which needed for the metadata.

This sounds like a good approach to link to pretty much any external backend (modulo the dlopen part).  I need to check with lawyer types before I can promise we'll be able to merge something like this into the main ceph.git, though!

Thanks-
sage

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Adding a proprietary key value store to CEPH

[Sage Weil]

Hi everyone,

I think there are good technical reasons to have a generic plugin loading 
framework.  New, experimental, or out of tree ObjectStore backends are 
one.  Another is Messenger, where it was recently noted that there is a 
compatibility issue between distro and OFED packages for IB verbs (used by 
the new XioMessenger).

My preferred approach here is a simple structure around the existing C++ 
abstract API and factory function pattern we're using for ObjectStore, 
KeyValueDB, and Messenger.  C++ ABI compatibility is fragile, so plugins 
will need to take care to use compatible build environments, but IMO that 
is a small price to pay for the minimal effort required (versus revamping 
all of these interfaces using dumbed-down C APIs).

This was discussed in some detail during the last CDS, but I admit I 
wasn't able to follow all of the points.  If I'm missing something 
important, please chime in!

Varada, I'm happy to go over in detail what I have in mind.  (I *think* it 
will be quite simple; hopefully I'm not missing any gotchas.)


As for the specific question of linking to a proprietary plugin using this 
mechanism, I am not a lawyer and in any case don't have any legal opinion 
to express.

I can say that my intention and expectation when originally choosing LGPL 
was that non-GPL code could exist both above and below Ceph in the stack.

It is also my opinion now, given everything I have seen over the last ten 
years, is that any closed source or out of tree backend is likely to be 
technically crippled by that decision and will prove to be more difficult 
to support.  Solutions are easier to develop, test, and support when they 
are open source, and I challenge organizations who feel they need to 
protect their secret sauce to get over themselves and focus on making the 
best possible solution for their customers.  For example, supporting an 
OSD backend based on f2fs (which is upstream in Linux) would make any 
testing and bug discussions with Ceph engineers--whomever they are 
employed by--much easier than with a closed library.  For what it's worth, 
given what I've seen so far from SanDisk in terms of their expertise and 
open approach to date with Ceph, the protection of IP around a key/value 
interfaces for their flash does not strike me as critical to success.  :)

Thanks!
sage