RGW Multisite delete wierdness

[Abhishek Lekshmanan <abhishek@suse.com>]

Trying deleting objects & buckets from a secondary zone in a RGW
multisite configuration leads to some wierdness:

1. On deleting an object and the bucket immediately will mostly lead to
object and bucket getting deleted in the secondary zone, but since we
forward the bucket deletion to master only after we delete in secondary
it will fail with 409 (BucketNotEmpty) and gets reraised as a 500 to the
client. This _seems_ simple enough to fix if we forward the bucket
deletion request to master zone before attempting deletion locally,
(issue: http://tracker.ceph.com/issues/15540, possible fix: https://github.com/ceph/ceph/pull/8655)

2. Deletion of objects themselves: deletion of objects themselves seems
to be a bit racy, deleting an object on a secondary zone succeeds,
listing the bucket seems to show an empty list, but gets populated with
the object again sometimes (this time with a newer timestamp), this is
not always guaranteed to be reproduce, but I've seen this often with
multipart uploads, as an eg:

$ s3 -u list test-mp
                       Key                             Last Modified      Size
--------------------------------------------------  --------------------  -----
test.img                                            2016-04-19T13:00:17Z    40M
$ s3 -u delete test-mp/test.img
$ s3 -u list test-mp
                       Key                             Last Modified      Size
--------------------------------------------------  --------------------  -----
test.img                                            2016-04-19T13:00:45Z    40M
$ s3 -u delete test-mp/test.img # wait for a  min
$ s3 -us list test-mp
--------------------------------------------------  --------------------  -----
test.img                                            2016-04-19T13:01:52Z    40M


Mostly seeing log entries of this form in both the cases ie. where
delete object seems to be successfully delete in both master and
secondary zone and the case where it succeeds in master and fails in
secondary :

20 parsed entry: id=00000000027.27.2 iter->object=foo iter->instance= name=foo instance= ns=
20 [inc sync] skipping object: dkr:d8e0ec3d-b3da-43f8-a99b-38a5b4941b6f.14113.2:-1/foo: non-complete operation
20 parsed entry: id=00000000028.28.2 iter->object=foo iter->instance= name=foo instance= ns=
20 [inc sync] skipping object: dkr:d8e0ec3d-b3da-43f8-a99b-38a5b4941b6f.14113.2:-1/foo: canceled operation

Any ideas on this?

--
Abhishek