Java bindings for RADOS and RBD

Java bindings for RADOS and RBD

[Wido den Hollander]

Hi all,

For the Apache CloudStack project I'm working on implementing the 
snapshotting and cloning features for RBD.

We (the project) want to stay away from invoking binaries like 'rbd' or 
'qemu-img' in the background since parsing the output and handling exit 
statuses can be hard to deal with.

So work around this I started working on Java JNA [0] bindings for 
librados and librbd [1]

The reason to use JNA is that it allows us to simply drop the bindings 
and run them without having them compiled against the librados or librbd 
headers.

The bindings are currently aimed at providing the functionality which 
CloudStack needs, which is mainly focused at RBD, but I've also 
implemented multiple RADOS operations.

I've chosen to implement both RADOS and RBD into the same package and 
using com.ceph.rados and com.ceph.rbd for as the package name. It could 
be splitted into different projects, but I think that won't benefit 
anybody. Having it all in one package seem easy, since RBD needs a RADOS 
IoCTX to work anyway.

I'd like to get some feedback on the bindings about the way they are 
now. They are still work-in-progress, but my Unit Testing shows me they 
are in a reasonable shape already.

Once the bindings reach a more mature state (which probably isn't that 
far away) I'd like to donate them to the Ceph project so they can live 
on Ceph's Github page instead of my personal one.

Getting them into Maven Central [2] won't be that easy, so I'd like to 
request space for that on ceph.com, for example ceph.com/download/maven 
and users will have to manually add that to their pom.xml configuration.

The request for a Maven repository is a bit selfish, since CloudStack 
uses Maven to build and during building it will need some place where it 
can fetch the artifacts.

So that's it for now, feedback and suggestions are more then welcome! 
(Patches / pull requests ditto!)

Related issues in the tracker:
- #4231
- #4232

[0]: https://github.com/twall/jna
[1]: https://github.com/wido/rados-java
[2]: 
http://maven.apache.org/guides/mini/guide-central-repository-upload.html

-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Fwd: Java bindings for RADOS and RBD

[Noah Watkins]

Resending. HTML+vger issue.

---------- Forwarded message ----------
From: Noah Watkins <noah.watkins@inktank.com>
Date: Thu, May 30, 2013 at 12:59 PM
Subject: Re: Java bindings for RADOS and RBD
To: Wido den Hollander <wido@42on.com>
Cc: ceph-devel <ceph-devel@vger.kernel.org>


On Mon, May 6, 2013 at 8:21 AM, Wido den Hollander <wido@42on.com> wrote:
>
>
> The reason to use JNA is that it allows us to simply drop the bindings and run them without having them compiled against the librados or librbd headers.


Nice. The JNA stuff is very easy to use. We originally looked at it
for the CephFS bindings, but there was some concern about dependencies
and licensing w.r.t. Hadoop.

>
> I've chosen to implement both RADOS and RBD into the same package and using com.ceph.rados and com.ceph.rbd for as the package name. It could be splitted into different projects, but I think that won't benefit anybody. Having it all in one package seem easy, since RBD needs a RADOS IoCTX to work anyway.


Cool. We have the com.ceph.fs and com.ceph.crush namespaces right now.

>
> I'd like to get some feedback on the bindings about the way they are now. They are still work-in-progress, but my Unit Testing shows me they are in a reasonable shape already.


They look like a really good start. Here is some feedback in no
particular order.

1. Enforcing rados usage assumptions

Unlike with the C interface where users are expected to behave, we
want to avoid ever crashing the JVM. So, stuff like "what happens if I
create an IoCTX, then destroy the Rados object and use the IoCTX?"
comes to mind. I think this would correspond to the GC running
finalize on an out of scope Rados object.

2. Making IoCTX safer to use

I've used the library now to bring the IoCTX into a completely
separate RADOS project. Designing for that to be common would be
great. For instance, that may be _very_ common for users of
IoCTX.Exec() since they will have completely distinct libraries.

A first step might be to expose a constant/read-only pointer. I'm not
a JNA expert, but after reading a bit, it seems as though we can
subclass IoCTX from Structure or maybe PointerType to make IoCTX
behave.

3. Async

Getting one or two async wrappers put it in the library might help
reveal any challenges early on, even if the API coverage expands
slowly.


> Getting them into Maven Central [2] won't be that easy, so I'd like to request space for that on ceph.com, for example ceph.com/download/maven and users will have to manually add that to their pom.xml configuration.


This would be really nice. I dunno what people expect from Maven
projects that rely on native libraries. At least getting debian
packages would be a good step.

Re: Java bindings for RADOS and RBD

[edison su]

On Thu, May 30, 2013 at 1:01 PM, Noah Watkins <noah.watkins@inktank.com> wrote:
> Resending. HTML+vger issue.
>
> ---------- Forwarded message ----------
> From: Noah Watkins <noah.watkins@inktank.com>
> Date: Thu, May 30, 2013 at 12:59 PM
> Subject: Re: Java bindings for RADOS and RBD
> To: Wido den Hollander <wido@42on.com>
> Cc: ceph-devel <ceph-devel@vger.kernel.org>
>
>
> On Mon, May 6, 2013 at 8:21 AM, Wido den Hollander <wido@42on.com> wrote:
>>
>>
>> The reason to use JNA is that it allows us to simply drop the bindings and run them without having them compiled against the librados or librbd headers.
>
>
> Nice. The JNA stuff is very easy to use. We originally looked at it
> for the CephFS bindings, but there was some concern about dependencies
> and licensing w.r.t. Hadoop.
>
>>
>> I've chosen to implement both RADOS and RBD into the same package and using com.ceph.rados and com.ceph.rbd for as the package name. It could be splitted into different projects, but I think that won't benefit anybody. Having it all in one package seem easy, since RBD needs a RADOS IoCTX to work anyway.
>
>
> Cool. We have the com.ceph.fs and com.ceph.crush namespaces right now.
>
>>
>> I'd like to get some feedback on the bindings about the way they are now. They are still work-in-progress, but my Unit Testing shows me they are in a reasonable shape already.
>
>
> They look like a really good start. Here is some feedback in no
> particular order.
>
> 1. Enforcing rados usage assumptions
>
> Unlike with the C interface where users are expected to behave, we
> want to avoid ever crashing the JVM. So, stuff like "what happens if I
> create an IoCTX, then destroy the Rados object and use the IoCTX?"
> comes to mind. I think this would correspond to the GC running
> finalize on an out of scope Rados object.
>
> 2. Making IoCTX safer to use
>
> I've used the library now to bring the IoCTX into a completely
> separate RADOS project. Designing for that to be common would be
> great. For instance, that may be _very_ common for users of
> IoCTX.Exec() since they will have completely distinct libraries.
>
> A first step might be to expose a constant/read-only pointer. I'm not
> a JNA expert, but after reading a bit, it seems as though we can
> subclass IoCTX from Structure or maybe PointerType to make IoCTX
> behave.
>
> 3. Async
>
> Getting one or two async wrappers put it in the library might help
> reveal any challenges early on, even if the API coverage expands
> slowly.

+1 on the Async support.

>
>
>> Getting them into Maven Central [2] won't be that easy, so I'd like to request space for that on ceph.com, for example ceph.com/download/maven and users will have to manually add that to their pom.xml configuration.
>
>
> This would be really nice. I dunno what people expect from Maven
> projects that rely on native libraries. At least getting debian
> packages would be a good step.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Best Regards,
Edison