a boto-works test for tabled?

a boto-works test for tabled?

[Jeff Garzik]

Looking at the 'python-boto' package in Fedora, I see it ships with 
boto's basic test suite.  Notably, it ships with a test module that 
exercises boto's S3 library routines.

Is there anyone that would be interested in copying (or directly use) 
/usr/lib/python2.6/site-packages/boto/tests/test_s3connection.py as a 
tabled "boto-works" test?  According to the boto author, boto should 
accept non-Amazon hostnames, which is the only requirement outside of 
Amazon AWS specifications that tabled has.

I am bloody awful at python, really don't know it well at all.  But to 
anyone who knows python, this will probably take all of an hour, from 
start to "make distcheck".  All anyone needs to do is create a python 
script 'test/boto-works' for tabled, which does exit(0) on success and 
exit(1) upon test failure.  "make check" or "make distcheck" initiate 
tabled's testsuite.

This would add a python-boto BuildRequires in tabled's rpm build, but I 
don't think that is a major issue.

	Jeff

Re: a boto-works test for tabled?

[Jeff Darcy]

On 12/18/2009 05:14 AM, Jeff Garzik wrote:
> Is there anyone that would be interested in copying (or directly use) 
> /usr/lib/python2.6/site-packages/boto/tests/test_s3connection.py as a 
> tabled "boto-works" test?  According to the boto author, boto should 
> accept non-Amazon hostnames, which is the only requirement outside of 
> Amazon AWS specifications that tabled has.

Boto can accept non-Amazon hostnames, but there's a bit of a trick to
making it work with tabled.  As of September 10, this was the magic formula.

>>> import boto
>>> import logging
>>> boto.set_file_logger("boto","/tmp/boto.out",logging.DEBUG)
>>> from boto.s3 import Connection
>>> x = Connection("foo","bar",host="localhost4",port=18080,is_secure=False,debug=99,calling_format=boto.s3.connection.OrdinaryCallingFormat())
>>> x.get_all_buckets()

The magic is in the calling_format part, which took a little while to
puzzle through.  We should probably consider supporting the default
calling format to make use of boto-based tools easier.

Re: a boto-works test for tabled?

[Pete Zaitcev]

On Fri, 18 Dec 2009 10:57:45 -0500
Jeff Darcy <jdarcy@redhat.com> wrote:

> On 12/18/2009 05:14 AM, Jeff Garzik wrote:

> > Is there anyone that would be interested in copying (or directly use) 
> > /usr/lib/python2.6/site-packages/boto/tests/test_s3connection.py as a 
> > tabled "boto-works" test?  According to the boto author, boto should 
> > accept non-Amazon hostnames, which is the only requirement outside of 
> > Amazon AWS specifications that tabled has.
> 
> Boto can accept non-Amazon hostnames, but there's a bit of a trick to
> making it work with tabled.  As of September 10, this was the magic formula.

> >>> x = Connection("foo","bar",host="localhost4",port=18080,is_secure=False,debug=99,calling_format=boto.s3.connection.OrdinaryCallingFormat())
> >>> x.get_all_buckets()
> 
> The magic is in the calling_format part, which took a little while to
> puzzle through.  We should probably consider supporting the default
> calling format to make use of boto-based tools easier.

This sounds strange on 2 counts.

1. I fixed the calling format issue long time ago. Mind that you have
to add a wildcard for it to work:

; Default port is 8081, we override it with -p 4499 in /etc/sysconfig/cld
_cld._udp       IN      SRV     10 50 4499 hitlain
_cld._udp       IN      SRV     10 50 4499 elanor
;
; Intel no-name experimental box, wildcard is for tabled
hitlain         IN      A       192.168.128.2
                IN      AAAA    fec0:0:0:1:0:0:c0a8:8002
*.hitlain.zaitcev.lan. IN A     192.168.128.2
                       IN AAAA  fec0:0:0:1:0:0:c0a8:8002

2. I see you using the port parameter, but when I tried that, Boto ignored
port and continued to use 80. The code formed a string URL without port
and then passed that to some HTTP libraries.

Since I have no clue about Python, I had trouble making their standard
testing harness to work, so I just copied the test into a local file,
and appended an invocation like this:

from boto.s3.connection import S3Connection
from boto.s3.connection import OrdinaryCallingFormat, SubdomainCallingFormat
from boto.exception import S3PermissionsError

# class S3ConnectionTest (unittest.TestCase):
class S3ConnectionTest:
    def test_1_basic(self):
        c = S3Connection(aws_access_key_id="testuser",
                         aws_secret_access_key="testpass",
                         is_secure=False,
                         host="hitlain.zaitcev.lan",
                         debug=1,
                         calling_format=SubdomainCallingFormat())
......................

t = S3ConnectionTest()
t.test_1_basic()

-- Pete

Re: a boto-works test for tabled?

[Pete Zaitcev]

On Fri, 18 Dec 2009 10:57:45 -0500
Jeff Darcy <jdarcy@redhat.com> wrote:

> Boto can accept non-Amazon hostnames, but there's a bit of a trick to
> making it work with tabled.  As of September 10, this was the magic formula.

I see what happened, the fix was later than 9/10:

http://git.kernel.org/?p=daemon/distsrv/tabled.git;a=commitdiff;h=f9d8b1229026d77efaf9b58d2145b5fd097aefd7;hp=e1c9069b3604e9c9e2946db80101d456598fef82

-- Pete

Re: a boto-works test for tabled?

[Jeff Darcy]

On 12/18/2009 12:09 PM, Pete Zaitcev wrote:
> 1. I fixed the calling format issue long time ago. Mind that you have
> to add a wildcard for it to work:
> 
> ; Default port is 8081, we override it with -p 4499 in /etc/sysconfig/cld
> _cld._udp       IN      SRV     10 50 4499 hitlain
> _cld._udp       IN      SRV     10 50 4499 elanor
> ;
> ; Intel no-name experimental box, wildcard is for tabled
> hitlain         IN      A       192.168.128.2
>                 IN      AAAA    fec0:0:0:1:0:0:c0a8:8002
> *.hitlain.zaitcev.lan. IN A     192.168.128.2
>                        IN AAAA  fec0:0:0:1:0:0:c0a8:8002

Yes, if you're willing/able to make DNS changes then that probably
works.  I suggest that it's an impediment to testing in most environments.

> 2. I see you using the port parameter, but when I tried that, Boto ignored
> port and continued to use 80. The code formed a string URL without port
> and then passed that to some HTTP libraries.

That seems odd to me.  Maybe it's a difference between the two
calling-format methods?

Re: a boto-works test for tabled?

[Jeff Garzik]

On 12/18/2009 12:09 PM, Pete Zaitcev wrote:
> Since I have no clue about Python, I had trouble making their standard
> testing harness to work, so I just copied the test into a local file,
> and appended an invocation like this:
>
> from boto.s3.connection import S3Connection
> from boto.s3.connection import OrdinaryCallingFormat, SubdomainCallingFormat
> from boto.exception import S3PermissionsError
>
> # class S3ConnectionTest (unittest.TestCase):
> class S3ConnectionTest:
>      def test_1_basic(self):
>          c = S3Connection(aws_access_key_id="testuser",
>                           aws_secret_access_key="testpass",
>                           is_secure=False,
>                           host="hitlain.zaitcev.lan",
>                           debug=1,
>                           calling_format=SubdomainCallingFormat())
> ......................
>
> t = S3ConnectionTest()
> t.test_1_basic()

Neat.  Does that return the desired exit status for our tests?

	Jeff

Re: a boto-works test for tabled?

[Jeff Garzik]

On 12/18/2009 12:16 PM, Jeff Darcy wrote:
> On 12/18/2009 12:09 PM, Pete Zaitcev wrote:
>> 1. I fixed the calling format issue long time ago. Mind that you have
>> to add a wildcard for it to work:
>>
>> ; Default port is 8081, we override it with -p 4499 in /etc/sysconfig/cld
>> _cld._udp       IN      SRV     10 50 4499 hitlain
>> _cld._udp       IN      SRV     10 50 4499 elanor
>> ;
>> ; Intel no-name experimental box, wildcard is for tabled
>> hitlain         IN      A       192.168.128.2
>>                  IN      AAAA    fec0:0:0:1:0:0:c0a8:8002
>> *.hitlain.zaitcev.lan. IN A     192.168.128.2
>>                         IN AAAA  fec0:0:0:1:0:0:c0a8:8002
>
> Yes, if you're willing/able to make DNS changes then that probably
> works.  I suggest that it's an impediment to testing in most environments.

Certainly an impediment in ours...  We have all kinds of hackery to make 
sure that "make distcheck" works for each package.

I have even considered shipping a DNS config, and installing a DNS 
server on an alternate port...  but not too many setups permit one to 
override the DNS query port or otherwise permit substitution of an 
alternate DNS service.

	Jeff