s3-tests development

s3-tests development

[Andrew Gaul]

Hello cephalopods, I use s3-tests to test S3Proxy[1], Apache jclouds,
and an internal project.  While s3-tests has good functionality as it
exists, the project has not progressed much over the last six months.  I
have submitted over 20 pull requests to fix incorrect tests and for
additional test coverage but most remain unreviewed[2].  Further
s3-tests remains biased towards Ceph and not the AWS de facto standard,
failing to run against the latter due to TooManyBuckets failures[3].
Finally some features like V4 signature support[4] will require more
extensive changes.  We are at-risk of diverging; how can we best move
forward together?

[1] https://github.com/andrewgaul/s3proxy
[2] https://github.com/ceph/s3-tests/pulls?q=is%3Aopen+is%3Apr
[3] https://github.com/ceph/s3-tests/issues/25
[4] https://github.com/ceph/s3-tests/issues/35

-- 
Andrew Gaul
http://gaul.org/

Re: s3-tests development

[M Ranga Swami Reddy]

Can you please share the pull requests urls..we can do quick review..

Thanks
Swami

On Fri, Feb 27, 2015 at 8:07 AM, Andrew Gaul <andrew@gaul.org> wrote:
> Hello cephalopods, I use s3-tests to test S3Proxy[1], Apache jclouds,
> and an internal project.  While s3-tests has good functionality as it
> exists, the project has not progressed much over the last six months.  I
> have submitted over 20 pull requests to fix incorrect tests and for
> additional test coverage but most remain unreviewed[2].  Further
> s3-tests remains biased towards Ceph and not the AWS de facto standard,
> failing to run against the latter due to TooManyBuckets failures[3].
> Finally some features like V4 signature support[4] will require more
> extensive changes.  We are at-risk of diverging; how can we best move
> forward together?
>
> [1] https://github.com/andrewgaul/s3proxy
> [2] https://github.com/ceph/s3-tests/pulls?q=is%3Aopen+is%3Apr
> [3] https://github.com/ceph/s3-tests/issues/25
> [4] https://github.com/ceph/s3-tests/issues/35
>
> --
> Andrew Gaul
> http://gaul.org/
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: s3-tests development

[Yehuda Sadeh-Weinraub]

----- Original Message -----
> From: "Andrew Gaul" <andrew@gaul.org>
> To: ceph-devel@vger.kernel.org
> Sent: Thursday, February 26, 2015 6:37:35 PM
> Subject: s3-tests development
> 
> Hello cephalopods, I use s3-tests to test S3Proxy[1], Apache jclouds,
> and an internal project.  While s3-tests has good functionality as it
> exists, the project has not progressed much over the last six months.  I
> have submitted over 20 pull requests to fix incorrect tests and for
> additional test coverage but most remain unreviewed[2].  Further

Right. We do need to be more responsive. The main reason we took our time was that these tests are used for the ceph nightly qa tests, and any changes in these that exposes new incompatibility will fail these. We'd rather first have the issue fixed, then merge the change. An alternative way to doing it is to open a ceph tracker issue about the incompatibility, mark the test as 'fails_on_rgw', in which case we could merge it immediately.

> s3-tests remains biased towards Ceph and not the AWS de facto standard,
> failing to run against the latter due to TooManyBuckets failures[3].

Not sure what's the appropriate way to attack this one. Maybe the create_bucket function could keep a list of created buckets and then remove them if encountering this error using lru.

> Finally some features like V4 signature support[4] will require more
> extensive changes.  We are at-risk of diverging; how can we best move
> forward together?

We'd be happy with more tests, and with more features tested even if these weren't implemented yet. It can later help with the development of these features. E.g., I looked a few weeks back at v4 signatures, and having such a test would have helped. The only thing we need though would be a way to easily disable such tests. So adding 'fails_on_rgw', or some other way to detect would help a lot.

Thanks,
Yehuda

> 
> [1] https://github.com/andrewgaul/s3proxy
> [2] https://github.com/ceph/s3-tests/pulls?q=is%3Aopen+is%3Apr
> [3] https://github.com/ceph/s3-tests/issues/25
> [4] https://github.com/ceph/s3-tests/issues/35
> 

Re: s3-tests development

[Alfredo Deza]

----- Original Message -----
From: "Yehuda Sadeh-Weinraub" <yehuda@redhat.com>
To: "Andrew Gaul" <andrew@gaul.org>
Cc: ceph-devel@vger.kernel.org, "Alfredo Deza" <adeza@redhat.com>
Sent: Friday, February 27, 2015 12:11:45 PM
Subject: Re: s3-tests development



----- Original Message -----
> From: "Andrew Gaul" <andrew@gaul.org>
> To: ceph-devel@vger.kernel.org
> Sent: Thursday, February 26, 2015 6:37:35 PM
> Subject: s3-tests development
> 
> Hello cephalopods, I use s3-tests to test S3Proxy[1], Apache jclouds,
> and an internal project.  While s3-tests has good functionality as it
> exists, the project has not progressed much over the last six months.  I
> have submitted over 20 pull requests to fix incorrect tests and for
> additional test coverage but most remain unreviewed[2].  Further

Right. We do need to be more responsive. The main reason we took our time was that these tests are used for the ceph nightly qa tests, and any changes in these that exposes new incompatibility will fail these. We'd rather first have the issue fixed, then merge the change. An alternative way to doing it is to open a ceph tracker issue about the incompatibility, mark the test as 'fails_on_rgw', in which case we could merge it immediately.

> s3-tests remains biased towards Ceph and not the AWS de facto standard,
> failing to run against the latter due to TooManyBuckets failures[3].

I have started some work to have a much cleaner way to identify documented API vs actual tests (making it clear where new tests should go) and to be able
to have leaner tests (vs. the current approach that stacks decorators).

This will also mean that we would move away from using the current test runner as it doesn't allow for such abstractions (in favor of py.test).

It will take some time to accomplish this, however we do still think that new tests and fixes should still be submitted while the new implementation is being worked on.

Not sure what's the appropriate way to attack this one. Maybe the create_bucket function could keep a list of created buckets and then remove them if encountering this error using lru.

> Finally some features like V4 signature support[4] will require more
> extensive changes.  We are at-risk of diverging; how can we best move
> forward together?

We'd be happy with more tests, and with more features tested even if these weren't implemented yet. It can later help with the development of these features. E.g., I looked a few weeks back at v4 signatures, and having such a test would have helped. The only thing we need though would be a way to easily disable such tests. So adding 'fails_on_rgw', or some other way to detect would help a lot.

Thanks,
Yehuda

> 
> [1] https://github.com/andrewgaul/s3proxy
> [2] https://github.com/ceph/s3-tests/pulls?q=is%3Aopen+is%3Apr
> [3] https://github.com/ceph/s3-tests/issues/25
> [4] https://github.com/ceph/s3-tests/issues/35
> 

Re: s3-tests development

[Andrew Gaul]

On Fri, Feb 27, 2015 at 12:11:45PM -0500, Yehuda Sadeh-Weinraub wrote:
> Andrew Gaul wrote:
> > While s3-tests has good functionality as it
> > exists, the project has not progressed much over the last six months.  I
> > have submitted over 20 pull requests to fix incorrect tests and for
> > additional test coverage but most remain unreviewed[2].
> 
> Right. We do need to be more responsive. The main reason we took our time was that these tests are used for the ceph nightly qa tests, and any changes in these that exposes new incompatibility will fail these. We'd rather first have the issue fixed, then merge the change. An alternative way to doing it is to open a ceph tracker issue about the incompatibility, mark the test as 'fails_on_rgw', in which case we could merge it immediately.

Instead of testing against master, perhaps you can tag a s3-tests 1.0.0
release and Ceph can use that?  Alternatively you can run your tests
against a specific commit hash.  In addition to running s3-tests against
Ceph, it would be good to run it against AWS, although we are blocked as
discussed below.

> > s3-tests remains biased towards Ceph and not the AWS de facto standard,
> > failing to run against the latter due to TooManyBuckets failures[3].
> 
> Not sure what's the appropriate way to attack this one. Maybe the create_bucket function could keep a list of created buckets and then remove them if encountering this error using lru.

jclouds uses an LRU scheme to recycle buckets between tests which works
well.  Alternatively s3-tests could remove buckets immediately after
each test although this has some tooling challenges discussed in the
issue.

> > Finally some features like V4 signature support[4] will require more
> > extensive changes.  We are at-risk of diverging; how can we best move
> > forward together?
> 
> We'd be happy with more tests, and with more features tested even if these weren't implemented yet. It can later help with the development of these features. E.g., I looked a few weeks back at v4 signatures, and having such a test would have helped. The only thing we need though would be a way to easily disable such tests. So adding 'fails_on_rgw', or some other way to detect would help a lot.

Annotating every failing test will not scale with additional providers;
I have almost 100 annotations for S3Proxy presently[1].  Could we
implement an external profile mechanism which tracks which tests should
or should not run?  Perhaps nosetests has a way to do this already?

[1] https://github.com/andrewgaul/s3-tests/commit/7891cb4d9a1cd6e6f4d8119f33c4f0bdb35348d3

-- 
Andrew Gaul
http://gaul.org/

Re: s3-tests development

[Sage Weil]

On Fri, 27 Feb 2015, Andrew Gaul wrote:
> On Fri, Feb 27, 2015 at 12:11:45PM -0500, Yehuda Sadeh-Weinraub wrote:
> > Andrew Gaul wrote:
> > > While s3-tests has good functionality as it
> > > exists, the project has not progressed much over the last six months.  I
> > > have submitted over 20 pull requests to fix incorrect tests and for
> > > additional test coverage but most remain unreviewed[2].
> > 
> > Right. We do need to be more responsive. The main reason we took our time was that these tests are used for the ceph nightly qa tests, and any changes in these that exposes new incompatibility will fail these. We'd rather first have the issue fixed, then merge the change. An alternative way to doing it is to open a ceph tracker issue about the incompatibility, mark the test as 'fails_on_rgw', in which case we could merge it immediately.
> 
> Instead of testing against master, perhaps you can tag a s3-tests 1.0.0
> release and Ceph can use that?  Alternatively you can run your tests
> against a specific commit hash.  In addition to running s3-tests against
> Ceph, it would be good to run it against AWS, although we are blocked as
> discussed below.

I think this makes sense.  I don't think we should hold back s3-tests 
development because radosgw isn't there yet.  Just as we have branches for 
ceph stable versions we can have one for ceph master and move that forward 
or rebase or annotate as we play catch-up with the tests...

sage

Re: s3-tests development

[Alfredo Deza]

----- Original Message -----
From: "Andrew Gaul" <andrew@gaul.org>
To: "Yehuda Sadeh-Weinraub" <yehuda@redhat.com>
Cc: ceph-devel@vger.kernel.org, "Alfredo Deza" <adeza@redhat.com>
Sent: Friday, February 27, 2015 1:26:23 PM
Subject: Re: s3-tests development

On Fri, Feb 27, 2015 at 12:11:45PM -0500, Yehuda Sadeh-Weinraub wrote:
> Andrew Gaul wrote:
> > While s3-tests has good functionality as it
> > exists, the project has not progressed much over the last six months.  I
> > have submitted over 20 pull requests to fix incorrect tests and for
> > additional test coverage but most remain unreviewed[2].
> 
> Right. We do need to be more responsive. The main reason we took our time was that these tests are used for the ceph nightly qa tests, and any changes in these that exposes new incompatibility will fail these. We'd rather first have the issue fixed, then merge the change. An alternative way to doing it is to open a ceph tracker issue about the incompatibility, mark the test as 'fails_on_rgw', in which case we could merge it immediately.

Instead of testing against master, perhaps you can tag a s3-tests 1.0.0
release and Ceph can use that?  Alternatively you can run your tests
against a specific commit hash.  In addition to running s3-tests against
Ceph, it would be good to run it against AWS, although we are blocked as
discussed below.

> > s3-tests remains biased towards Ceph and not the AWS de facto standard,
> > failing to run against the latter due to TooManyBuckets failures[3].
> 
> Not sure what's the appropriate way to attack this one. Maybe the create_bucket function could keep a list of created buckets and then remove them if encountering this error using lru.

jclouds uses an LRU scheme to recycle buckets between tests which works
well.  Alternatively s3-tests could remove buckets immediately after
each test although this has some tooling challenges discussed in the
issue.

> > Finally some features like V4 signature support[4] will require more
> > extensive changes.  We are at-risk of diverging; how can we best move
> > forward together?
> 
> We'd be happy with more tests, and with more features tested even if these weren't implemented yet. It can later help with the development of these features. E.g., I looked a few weeks back at v4 signatures, and having such a test would have helped. The only thing we need though would be a way to easily disable such tests. So adding 'fails_on_rgw', or some other way to detect would help a lot.

Annotating every failing test will not scale with additional providers;
I have almost 100 annotations for S3Proxy presently[1].  Could we
implement an external profile mechanism which tracks which tests should
or should not run?  Perhaps nosetests has a way to do this already?

Like I mentioned, we are working to port these to py.test with a re-organization included that should allow to configure what tests are run
and how failures are treated.



[1] https://github.com/andrewgaul/s3-tests/commit/7891cb4d9a1cd6e6f4d8119f33c4f0bdb35348d3

-- 
Andrew Gaul
http://gaul.org/