RadosGW objects to Rados object mapping

[Abhishek L <abhishek.lekshmanan@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

Hi,

I'm trying to understand the internals of RadosGW, on how
buckets/containers, objects are mapped back to rados objects. I couldn't
find any docs, however a previous mailing list discussion[1] explained
how an S3/Swift objects are cut into rados objects and about manifests. I was
able to construct back a file uploaded to RadosGW by getting the rados
objects by using the manifest to figure out the rados object names. 
For eg:
```
# random.txt is an 8 MB text file
[r@ra:~/ceph/src]$ s3 -us put my-first-bucket/random filename=random.txt 
[r@ra:~/ceph/src]$ ./radosgw-admin object stat --bucket=my-first-bucket --object=random  | grep prefix 
      "prefix": "._op2xmptte2DD7z3_9EjQKgmmRcWRWL_",

```

And then getting the objects via rados and joining back

```
[r@ra:~/ceph/src]$ ./rados --pool .rgw.buckets ls | grep _op2xm
default.4124.1__shadow_._op2xmptte2DD7z3_9EjQKgmmRcWRWL_2
default.4124.1__shadow_._op2xmptte2DD7z3_9EjQKgmmRcWRWL_1
[r@ra:~/ceph/src]$ ./rados get default.4124.1_random random.part0 --pool .rgw.buckets
[r@ra:~/ceph/src]$ ./rados get default.4124.1__shadow_._op2xmptte2DD7z3_9EjQKgmmRcWRWL_1 random.part1 --pool .rgw.buckets
[r@ra:~/ceph/src]$ ./rados get default.4124.1__shadow_._op2xmptte2DD7z3_9EjQKgmmRcWRWL_2 random.part2 --pool .rgw.buckets

# Now join the objects back 
[r@ra:~/ceph/src]$ cat random.part0 random.part1 random.part2 > random.rados.txt
[r@ra:~/ceph/src]$ diff random.txt random.rados.txt 
```

I'm trying to find similiar information on how radosgw ends up storing
the buckets & metadata into rados objects, what information is
contained within them and how they are updated when say an object is
added etc. I was able to find the bucket name & bucket meta data being
stored in .rgw pool, but not sure how the bucket knows the objects it
has or buckets owned by user etc.

[1] https://www.mail-archive.com/ceph-devel@vger.kernel.org/msg19747.html 

Thanks
-- 
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]