Metadata queries on Ceph/RADOS

Metadata queries on Ceph/RADOS

[Atish Kathpal]

Hello

Does RADOS support metadata queries? I have gone through librados and
the "rados" command line tool, and there doesn't seem to be a way for
me to say, query the object store saying: "Give me all objects with
Xattr Video = 1". The only way I can think of is to write some of my
own code using librados.

Any pointers if the system has a metadata service or something that I
can query?

(I have seen few diagrams showing a metadata cluster as part of Ceph
which supports metadata operations, how can I leverage that? reference
diagram: http://www.usenix.org/event/osdi06/tech/full_papers/weil/weil_html/fig/overview.png)

 Regards
Atish
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Metadata queries on Ceph/RADOS

[Josh Durgin]

On 11/22/2011 12:11 AM, Atish Kathpal wrote:
> Hello
>
> Does RADOS support metadata queries? I have gone through librados and
> the "rados" command line tool, and there doesn't seem to be a way for
> me to say, query the object store saying: "Give me all objects with
> Xattr Video = 1". The only way I can think of is to write some of my
> own code using librados.

That's right, RADOS operates on a single-object transaction level. 
There's no way for it to support consistent reads of multiple objects 
without rewriting much of the OSD - it's not designed for that. For your 
use case, you could store your metadata in a trivial map (tmap) object 
using librados' tmap_* functions. This is what radosgw uses for storing 
which objects are in which buckets. These are slow for large mappings, 
but we have plans to fix that  (http://tracker.newdream.net/issues/1571).

> Any pointers if the system has a metadata service or something that I
> can query?
>
> (I have seen few diagrams showing a metadata cluster as part of Ceph
> which supports metadata operations, how can I leverage that? reference
> diagram: http://www.usenix.org/event/osdi06/tech/full_papers/weil/weil_html/fig/overview.png)

The metadata cluster is for the Ceph distruted file system, which is 
another layer on top of RADOS.

Re: Metadata queries on Ceph/RADOS

[Atish Kathpal]

Thank you for the insights.

Atish

On Tue, Nov 22, 2011 at 10:39 PM, Gregory Farnum
<gregory.farnum@dreamhost.com> wrote:
>
>
> On Tuesday, November 22, 2011, Atish Kathpal <atish.kathpal@gmail.com>
> wrote:
>> Hello
>>
>> Does RADOS support metadata queries?
> Nope. RADOS is not a database of any kind, sorry. The only metadata service
> is that used by te Ceph filesystem, and it can't do what you're after.
> I have gone through librados and
>> the "rados" command line tool, and there doesn't seem to be a way for
>> me to say, query the object store saying: "Give me all objects with
>> Xattr Video = 1". The only way I can think of is to write some of my
>> own code using librados.
>>
>> Any pointers if the system has a metadata service or something that I
>> can query?
> You could see of the object classes and map objects let you do what you
> want. I think we'll need to make them a bit more powerful before they're
> helpful for you, though. :/
>
>>
>> (I have seen few diagrams showing a metadata cluster as part of Ceph
>> which supports metadata operations, how can I leverage that? reference
>> diagram:
>> http://www.usenix.org/event/osdi06/tech/full_papers/weil/weil_html/fig/overview.png)
>>
>>  Regards
>> Atish
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html