* Access Dienied for bucket upload - 403 code
@ 2012-09-11 14:28 Sławomir Skowron
2012-09-11 16:45 ` Yehuda Sadeh
0 siblings, 1 reply; 16+ messages in thread
From: Sławomir Skowron @ 2012-09-11 14:28 UTC (permalink / raw)
To: ceph-devel
Every acl operation ending with 403 in PUT.
~# s3 -u test oc
Bucket Status
-------------------------------------------------------- --------------------
oc Access Denied
Anyone know why, and how to enable this bucket ?? Now i have problems
with cluster, because there is no way to upload new file
~# s3 -u getacl oc
ERROR: ErrorAccessDenied
--
-----
Regards
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Access Dienied for bucket upload - 403 code 2012-09-11 14:28 Access Dienied for bucket upload - 403 code Sławomir Skowron @ 2012-09-11 16:45 ` Yehuda Sadeh 2012-09-11 16:48 ` Yehuda Sadeh 0 siblings, 1 reply; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 16:45 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: > Every acl operation ending with 403 in PUT. > > ~# s3 -u test oc > Bucket Status > -------------------------------------------------------- -------------------- > oc Access Denied > > Anyone know why, and how to enable this bucket ?? Now i have problems > with cluster, because there is no way to upload new file > > ~# s3 -u getacl oc > > ERROR: ErrorAccessDenied > User somehow lost bucket ownership (was it actually the owner?). Do you know how to reproduce the issue? any remaining logs? Try getting bucket info: # radosgw-admin bucket stats --bucket=oc If that doesn't fail and actually shows relevant info, try checking whether the user credentials match the s3 tool credentials. Thanks, Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 16:45 ` Yehuda Sadeh @ 2012-09-11 16:48 ` Yehuda Sadeh 2012-09-11 17:46 ` Sławomir Skowron 0 siblings, 1 reply; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 16:48 UTC (permalink / raw) To: Sławomir Skowron, ceph-devel On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: > On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >> Every acl operation ending with 403 in PUT. >> >> ~# s3 -u test oc >> Bucket Status >> -------------------------------------------------------- -------------------- >> oc Access Denied >> >> Anyone know why, and how to enable this bucket ?? Now i have problems >> with cluster, because there is no way to upload new file >> >> ~# s3 -u getacl oc >> >> ERROR: ErrorAccessDenied >> > > User somehow lost bucket ownership (was it actually the owner?). Do > you know how to reproduce the issue? any remaining logs? > > Try getting bucket info: > > # radosgw-admin bucket stats --bucket=oc > > If that doesn't fail and actually shows relevant info, try checking > whether the user credentials match the s3 tool credentials. > Oh, and thinking about it some more.. 'oc' is a too short name for a bucket (requires min of 3 chars). How did you create it? The failure may be related. Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 16:48 ` Yehuda Sadeh @ 2012-09-11 17:46 ` Sławomir Skowron 2012-09-11 20:41 ` Sławomir Skowron 0 siblings, 1 reply; 16+ messages in thread From: Sławomir Skowron @ 2012-09-11 17:46 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: > On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>> Every acl operation ending with 403 in PUT. >>> >>> ~# s3 -u test oc >>> Bucket Status >>> -------------------------------------------------------- -------------------- >>> oc Access Denied >>> >>> Anyone know why, and how to enable this bucket ?? Now i have problems >>> with cluster, because there is no way to upload new file >>> >>> ~# s3 -u getacl oc >>> >>> ERROR: ErrorAccessDenied >>> >> >> User somehow lost bucket ownership (was it actually the owner?). Do >> you know how to reproduce the issue? any remaining logs? >> >> Try getting bucket info: >> >> # radosgw-admin bucket stats --bucket=oc >> >> If that doesn't fail and actually shows relevant info, try checking >> whether the user credentials match the s3 tool credentials. >> > Oh, and thinking about it some more.. 'oc' is a too short name for a > bucket (requires min of 3 chars). How did you create it? The failure > may be related. Yes i made a shortcut of name :)) Right now every bucket in pool, are afected :~#radosgw-admin bucket stats --bucket=lvstest { "bucket": "lvstest", "pool": ".rgw.buckets", "id": "1142048.1", "marker": "1142048.1", "owner": "0", "usage": { "rgw.main": { "size_kb": 1, "size_kb_actual": 4, "num_objects": 1}}} :~# radosgw-admin bucket stats --bucket=ocdn { "bucket": "ocdn", "pool": ".rgw.buckets", "id": "4168.2", "marker": "4168.2", "owner": "0", "usage": { "rgw.main": { "size_kb": 513059717, "size_kb_actual": 516402364, "num_objects": 1606730}}} Credentials from radosgw-admin user info match that from clients requests. Every GET, PUT, HEAD using this credentials works fine, but only one operations does not work (403 from radosgw) - setting acl for object for a public-read. Setting canned acl with PUT for public-read from s3lib work good, but get/set acl failed. list bucket object works good, and list buckets via s3 client. Now i can't reproduce, but i will dig logs from radosgw, for related time, when this happend. Example 403 from radosgw log, before that PUT of object ends with 200: 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done req=0x1435980 http_status=403 ====== 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request req=0x13994c0 ===== 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing 2012-09-11 19:37:04.342971 7fb25d7fa700 10 s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg s->bucket=ocdn 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 2012-09-11 19:37:04.343005 7fb25d7fa700 20 SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg 2012-09-11 19:37:04.343006 7fb25d7fa700 20 REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg 2012-09-11 19:37:04.343007 7fb25d7fa700 20 DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 17:37:02 GMT 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting op 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb23c00b468 s->prefetch_data=0 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb23c00b468 s->prefetch_data=0 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb23c00b468 s->prefetch_data=0 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic test 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= 2012-09-11 19:37:04.347198 7fb25d7fa700 15 auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http status=403 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done req=0x13994c0 http_status=403 ====== 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 Interesting is that second cluster, which is replicated via s3 client, get the same issue, from that in production one. Is this can be generated by many error on radosgw layer ?? I will try to find beginning of this problems in logs. > > Yehuda -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 17:46 ` Sławomir Skowron @ 2012-09-11 20:41 ` Sławomir Skowron 2012-09-11 20:46 ` Yehuda Sadeh 2012-09-11 23:27 ` Yehuda Sadeh 0 siblings, 2 replies; 16+ messages in thread From: Sławomir Skowron @ 2012-09-11 20:41 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel And more logs: 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done req=0x141a650 http_status=403 ====== 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request req=0x139a3d0 ===== 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 ...skipping... 2012-09-11 22:23:44.530567 7faf0bf4f700 10 s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg s->bucket=ocdn 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= 2012-09-11 22:23:44.530595 7faf0bf4f700 20 SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530596 7faf0bf4f700 20 REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530598 7faf0bf4f700 20 DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 2012-09-11 22:23:44.530615 7faf0bf4f700 20 HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 2012-09-11 22:23:44.530616 7faf0bf4f700 20 HTTP_ACCEPT_ENCODING=gzip,deflate,sdch 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 2012-09-11 22:23:44.530618 7faf0bf4f700 20 HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting op 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading permissions 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 s->prefetch_data=0 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 s->prefetch_data=0 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 s->prefetch_data=0 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is not atomic, not appending atomic test 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: rctx=0x7fb01c0122c0 obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg state=0x7fb01c00a608 s->prefetch_data=1 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying op permissions 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for uid=anonymous mask=1 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for group=1 mask=1 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for uid=anonymous mask=16 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for group=1 mask=16 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http status=403 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done req=0x13995c0 http_status=403 ====== Good put, and 403 after: 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request req=0x139a3d0 ===== 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing 2012-09-11 22:37:34.677492 7fafddffb700 10 s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 s->bucket=ocdn 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 2012-09-11 22:37:34.677508 7fafddffb700 20 SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.677509 7fafddffb700 20 REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.677510 7fafddffb700 20 DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity 2012-09-11 22:37:34.677521 7fafddffb700 20 HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 20:37:46 GMT 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting op 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb00c004a18 s->prefetch_data=0 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb00c004a18 s->prefetch_data=0 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb00c004a18 s->prefetch_data=0 2012-09-11 22:37:34.680452 7fafddffb700 20 state for obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic test 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= 2012-09-11 22:37:34.681921 7fafddffb700 15 auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading permissions 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 s->prefetch_data=0 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 s->prefetch_data=0 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 s->prefetch_data=0 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is not atomic, not appending atomic test 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 2012-09-11 22:37:34.684786 7fafddffb700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying op permissions 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for uid=0 mask=2 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying op params 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing 2012-09-11 22:37:34.684831 7fafddffb700 15 supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 2012-09-11 22:37:34.684841 7fafddffb700 15 supplied_md5=ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: rctx=0x7fb00c007050 obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 state=0x7fb00c00bda8 s->prefetch_data=0 2012-09-11 22:37:34.686278 7fafddffb700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7fb00c00bda8 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http status=200 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done req=0x139a3d0 http_status=200 ====== 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request req=0x1368860 ===== 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing 2012-09-11 22:37:34.708402 7faec76c6700 10 s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 s->bucket=ocdn 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 2012-09-11 22:37:34.708440 7faec76c6700 20 SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.708441 7faec76c6700 20 REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.708441 7faec76c6700 20 DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 20:37:46 GMT 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting op 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb2a4007f18 s->prefetch_data=0 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb2a4007f18 s->prefetch_data=0 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7fb2a4007f18 s->prefetch_data=0 2012-09-11 22:37:34.711108 7faec76c6700 20 state for obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic test 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= 2012-09-11 22:37:34.712742 7faec76c6700 15 auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request 2012-09-11 22:37:34.712754 7faec76c6700 10 --> Status: 403 2012-09-11 22:37:34.712766 7faec76c6700 10 --> Content-Length: 78 2012-09-11 22:37:34.712769 7faec76c6700 10 --> Accept-Ranges: bytes 2012-09-11 22:37:34.712772 7faec76c6700 10 --> Content-type: application/xml 2012-09-11 22:37:34.712887 7faec76c6700 2 req 71188:0.004554:s3:PUT /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http status=403 2012-09-11 22:37:34.713093 7faec76c6700 1 ====== req done req=0x1368860 http_status=403 ====== On Tue, Sep 11, 2012 at 7:46 PM, Sławomir Skowron <szibis@gmail.com> wrote: > On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>>> Every acl operation ending with 403 in PUT. >>>> >>>> ~# s3 -u test oc >>>> Bucket Status >>>> -------------------------------------------------------- -------------------- >>>> oc Access Denied >>>> >>>> Anyone know why, and how to enable this bucket ?? Now i have problems >>>> with cluster, because there is no way to upload new file >>>> >>>> ~# s3 -u getacl oc >>>> >>>> ERROR: ErrorAccessDenied >>>> >>> >>> User somehow lost bucket ownership (was it actually the owner?). Do >>> you know how to reproduce the issue? any remaining logs? >>> >>> Try getting bucket info: >>> >>> # radosgw-admin bucket stats --bucket=oc >>> >>> If that doesn't fail and actually shows relevant info, try checking >>> whether the user credentials match the s3 tool credentials. >>> >> Oh, and thinking about it some more.. 'oc' is a too short name for a >> bucket (requires min of 3 chars). How did you create it? The failure >> may be related. > > Yes i made a shortcut of name :)) > > Right now every bucket in pool, are afected > > :~#radosgw-admin bucket stats --bucket=lvstest > { "bucket": "lvstest", > "pool": ".rgw.buckets", > "id": "1142048.1", > "marker": "1142048.1", > "owner": "0", > "usage": { "rgw.main": { "size_kb": 1, > "size_kb_actual": 4, > "num_objects": 1}}} > :~# radosgw-admin bucket stats --bucket=ocdn > { "bucket": "ocdn", > "pool": ".rgw.buckets", > "id": "4168.2", > "marker": "4168.2", > "owner": "0", > "usage": { "rgw.main": { "size_kb": 513059717, > "size_kb_actual": 516402364, > "num_objects": 1606730}}} > > Credentials from radosgw-admin user info match that from clients requests. > > Every GET, PUT, HEAD using this credentials works fine, but only one > operations does not work (403 from radosgw) - setting acl for object > for a public-read. Setting canned acl with PUT for public-read from > s3lib work good, but get/set acl failed. > > list bucket object works good, and list buckets via s3 client. > > Now i can't reproduce, but i will dig logs from radosgw, for related > time, when this happend. > > Example 403 from radosgw log, before that PUT of object ends with 200: > > 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done > req=0x1435980 http_status=403 ====== > 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 > 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty > 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request > req=0x13994c0 ===== > 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing > 2012-09-11 19:37:04.342971 7fb25d7fa700 10 > s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg > s->bucket=ocdn > 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read > 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read > 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER > 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl > 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT > 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= > 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 > 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 > 2012-09-11 19:37:04.343005 7fb25d7fa700 20 > SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg > 2012-09-11 19:37:04.343006 7fb25d7fa700 20 > REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg > 2012-09-11 19:37:04.343007 7fb25d7fa700 20 > DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg > 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 > 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 > 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 > 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 > 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= > 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 > 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no > 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 > 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity > 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 > 17:37:02 GMT > 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read > 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS > 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= > 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) > 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT > /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting > op > 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT > /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing > 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: > rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb23c00b468 s->prefetch_data=0 > 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: > rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb23c00b468 s->prefetch_data=0 > 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: > rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb23c00b468 s->prefetch_data=0 > 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for > obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic > test > 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 > 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): > dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg > 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: > 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= > 2012-09-11 19:37:04.347198 7fb25d7fa700 15 > auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= > 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 > 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request > 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 > 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 > 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes > 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml > 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT > /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http > status=403 > 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done > req=0x13994c0 http_status=403 ====== > 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 > > Interesting is that second cluster, which is replicated via s3 client, > get the same issue, from that in production one. Is this can be > generated by many error on radosgw layer ?? > > I will try to find beginning of this problems in logs. > >> >> Yehuda > > > > -- > ----- > Pozdrawiam > > Sławek "sZiBis" Skowron -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 20:41 ` Sławomir Skowron @ 2012-09-11 20:46 ` Yehuda Sadeh 2012-09-11 21:28 ` Sławomir Skowron 2012-09-11 23:27 ` Yehuda Sadeh 1 sibling, 1 reply; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 20:46 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: > And more logs: > > > 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done > req=0x141a650 http_status=403 ====== > 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 > 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty > 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request > req=0x139a3d0 ===== > 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing > 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> > 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER > 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= > 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET > 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= > 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= > 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= > 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ > 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ > 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ > 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 > 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 > 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 > 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 > 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 > ...skipping... > 2012-09-11 22:23:44.530567 7faf0bf4f700 10 > s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > s->bucket=ocdn > 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= > 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET > 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= > 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= > 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= > 2012-09-11 22:23:44.530595 7faf0bf4f700 20 > SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530596 7faf0bf4f700 20 > REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530598 7faf0bf4f700 20 > DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 > 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 > 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 > 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 > 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= > 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 > 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 > 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive > 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 > (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like > Gecko) Chrome/21.0.1180.89 Safari/537.1 > 2012-09-11 22:23:44.530615 7faf0bf4f700 20 > HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 2012-09-11 22:23:44.530616 7faf0bf4f700 20 > HTTP_ACCEPT_ENCODING=gzip,deflate,sdch > 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 > 2012-09-11 22:23:44.530618 7faf0bf4f700 20 > HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 > 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting > op > 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing > 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading > permissions > 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is > not atomic, not appending atomic test > 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 > 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: > bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 > 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 > 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 > obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > state=0x7fb01c00a608 s->prefetch_data=1 > 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying > op permissions > 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for > uid=anonymous mask=1 > 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found > 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for > group=1 mask=1 > 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found > 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions > id=anonymous owner=0 perm=0 > 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested > perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 > 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for > uid=anonymous mask=16 > 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found > 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for > group=1 mask=16 > 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 > 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions > id=anonymous owner=0 perm=0 > 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested > perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 > 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 > 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 > 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes > 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml > 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http > status=403 > 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done > req=0x13995c0 http_status=403 ====== > > Good put, and 403 after: > > 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request > req=0x139a3d0 ===== > 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing > 2012-09-11 22:37:34.677492 7fafddffb700 10 > s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > s->bucket=ocdn > 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= > 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT > 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain > 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 > 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 > 2012-09-11 22:37:34.677508 7fafddffb700 20 > SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677509 7fafddffb700 20 > REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677510 7fafddffb700 20 > DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 > 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 > 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 > 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 > 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= > 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 > 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 > 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity > 2012-09-11 22:37:34.677521 7fafddffb700 20 > HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== > 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue > 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 > 20:37:46 GMT > 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) > 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain > 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS > 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting > op > 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing > 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.680452 7fafddffb700 20 state for > obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic > test > 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 > 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): > dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: > 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.681921 7fafddffb700 15 > auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 > 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading > permissions > 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is > not atomic, not appending atomic test > 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 > 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: > bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 > 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 > 2012-09-11 22:37:34.684786 7fafddffb700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying > op permissions > 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for > uid=0 mask=2 > 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 > 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm > (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 > 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying > op params > 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing > 2012-09-11 22:37:34.684831 7fafddffb700 15 > supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== > 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 > 2012-09-11 22:37:34.684841 7fafddffb700 15 > supplied_md5=ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 > obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > state=0x7fb00c00bda8 s->prefetch_data=0 > 2012-09-11 22:37:34.686278 7fafddffb700 20 > prepare_atomic_for_write_impl: state is not atomic. > state=0x7fb00c00bda8 > 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 > 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 > 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes > 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 > 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml > 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http > status=200 > 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done > req=0x139a3d0 http_status=200 ====== > > > 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 > 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty > 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request > req=0x1368860 ===== > 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing > 2012-09-11 22:37:34.708402 7faec76c6700 10 > s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > s->bucket=ocdn > 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read > 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read > 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl > 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT > 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= > 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 > 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 > 2012-09-11 22:37:34.708440 7faec76c6700 20 > SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708441 7faec76c6700 20 > REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708441 7faec76c6700 20 > DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 > 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 > 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 > 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 > 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= > 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 > 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 > 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity > 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 > 20:37:46 GMT > 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read > 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS > 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= > 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) > 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting > op > 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing > 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.711108 7faec76c6700 20 state for > obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic > test > 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 > 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): > dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: > 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= > 2012-09-11 22:37:34.712742 7faec76c6700 15 > auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= > 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 > 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request The request itself fails to authorize. I think it's related to trying to do canned acl put_acl. Not sure if that's actually supposed to work. Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 20:46 ` Yehuda Sadeh @ 2012-09-11 21:28 ` Sławomir Skowron 2012-09-11 21:32 ` Yehuda Sadeh 0 siblings, 1 reply; 16+ messages in thread From: Sławomir Skowron @ 2012-09-11 21:28 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel :~# s3 -u put ocdn/test3 cannedAcl=public-read < /tmp/testdl :~# s3 -u get ocdn/test3 > /tmp/test3 :~# HEAD http://127.0.0.1/ocdn/test3 200 OK Connection: close Date: Tue, 11 Sep 2012 21:23:19 GMT Accept-Ranges: bytes ETag: "a241f32b8cf07f90d36b5199629b8829" Server: nginx Content-Length: 6713 Last-Modified: Tue, 11 Sep 2012 21:22:48 GMT Client-Date: Tue, 11 Sep 2012 21:23:19 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 put with cannedacl forpublic works on this object. 2012-09-11 23:22:48.288059 7f67bd722700 10 s->object=test3 s->bucket=ocdn 2012-09-11 23:22:48.288064 7f67bd722700 10 meta>> HTTP_X_AMZ_ACL=public-read 2012-09-11 23:22:48.288070 7f67bd722700 10 meta>> HTTP_X_AMZ_DATE=Tue, 11 Sep 2012 21:22:48 GMT 2012-09-11 23:22:48.288074 7f67bd722700 10 x>> x-amz-acl:public-read 2012-09-11 23:22:48.288076 7f67bd722700 10 x>> x-amz-date:Tue, 11 Sep 2012 21:22:48 GMT 2012-09-11 23:22:48.288080 7f67bd722700 20 FCGI_ROLE=RESPONDER 2012-09-11 23:22:48.288081 7f67bd722700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 23:22:48.288083 7f67bd722700 20 QUERY_STRING= 2012-09-11 23:22:48.288084 7f67bd722700 20 REQUEST_METHOD=PUT 2012-09-11 23:22:48.288085 7f67bd722700 20 CONTENT_TYPE= 2012-09-11 23:22:48.288086 7f67bd722700 20 CONTENT_LENGTH=6713 2012-09-11 23:22:48.288087 7f67bd722700 20 HTTP_CONTENT_LENGTH=6713 2012-09-11 23:22:48.288088 7f67bd722700 20 SCRIPT_NAME=/ocdn/test3 2012-09-11 23:22:48.288089 7f67bd722700 20 REQUEST_URI=/ocdn/test3 2012-09-11 23:22:48.288089 7f67bd722700 20 DOCUMENT_URI=/ocdn/test3 2012-09-11 23:22:48.288090 7f67bd722700 20 DOCUMENT_ROOT=/var/www 2012-09-11 23:22:48.288091 7f67bd722700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 23:22:48.288092 7f67bd722700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 23:22:48.288092 7f67bd722700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 23:22:48.288093 7f67bd722700 20 REMOTE_ADDR=127.0.0.1 2012-09-11 23:22:48.288095 7f67bd722700 20 REMOTE_PORT=40421 2012-09-11 23:22:48.288096 7f67bd722700 20 SERVER_ADDR=127.0.0.1 2012-09-11 23:22:48.288097 7f67bd722700 20 SERVER_PORT=80 2012-09-11 23:22:48.288098 7f67bd722700 20 SERVER_NAME= 2012-09-11 23:22:48.288099 7f67bd722700 20 REDIRECT_STATUS=200 2012-09-11 23:22:48.288099 7f67bd722700 20 RGW_SHOULD_LOG=no 2012-09-11 23:22:48.288100 7f67bd722700 20 HTTP_USER_AGENT=Mozilla/4.0 (Compatible; s3; libs3 2.0; Linux x86_64) 2012-09-11 23:22:48.288101 7f67bd722700 20 HTTP_HOST=s3.amazonaws.com 2012-09-11 23:22:48.288102 7f67bd722700 20 HTTP_ACCEPT=*/* 2012-09-11 23:22:48.288103 7f67bd722700 20 HTTP_AUTHORIZATION=AWS 73VF66Q2JH5PT5K7QJ3A:MuMc3TusY8KsVkYUZENVU+FLs6Q= 2012-09-11 23:22:48.288104 7f67bd722700 20 HTTP_X_AMZ_ACL=public-read 2012-09-11 23:22:48.288104 7f67bd722700 20 HTTP_X_AMZ_DATE=Tue, 11 Sep 2012 21:22:48 GMT 2012-09-11 23:22:48.288105 7f67bd722700 20 HTTP_EXPECT=100-continue 2012-09-11 23:22:48.288107 7f67bd722700 2 req 75477:0.000112:s3:PUT /ocdn/test3::getting op 2012-09-11 23:22:48.288111 7f67bd722700 2 req 75477:0.000116:s3:PUT /ocdn/test3:put_obj:authorizing 2012-09-11 23:22:48.288126 7f67bd722700 20 get_obj_state: rctx=0x7f6a88002120 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7f6a880021d8 s->prefetch_data=0 2012-09-11 23:22:48.290419 7f67bd722700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 23:22:48.290434 7f67bd722700 20 get_obj_state: rctx=0x7f6a88002120 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7f6a880021d8 s->prefetch_data=0 2012-09-11 23:22:48.292004 7f67bd722700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 23:22:48.292016 7f67bd722700 20 get_obj_state: rctx=0x7f6a88002120 obj=.users:73VF66Q2JH5PT5K7QJ3A state=0x7f6a880021d8 s->prefetch_data=0 2012-09-11 23:22:48.292023 7f67bd722700 20 state for obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic test 2012-09-11 23:22:48.292026 7f67bd722700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 23:22:48.293258 7f67bd722700 20 rados->read r=0 bl.length=231 2012-09-11 23:22:48.293312 7f67bd722700 10 get_canon_resource(): dest=/ocdn/test3 2012-09-11 23:22:48.293316 7f67bd722700 10 auth_hdr: 2012-09-11 23:22:48.293397 7f67bd722700 15 b64=MuMc3TusY8KsVkYUZENVU+FLs6Q= 2012-09-11 23:22:48.293403 7f67bd722700 15 auth_sign=MuMc3TusY8KsVkYUZENVU+FLs6Q= 2012-09-11 23:22:48.293405 7f67bd722700 15 compare=0 2012-09-11 23:22:48.293408 7f67bd722700 2 req 75477:0.005412:s3:PUT /ocdn/test3:put_obj:reading permissions 2012-09-11 23:22:48.293420 7f67bd722700 20 get_obj_state: rctx=0x7f6a8800a6f0 obj=.rgw:ocdn state=0x7f6a8800b3b8 s->prefetch_data=0 2012-09-11 23:22:48.294866 7f67bd722700 20 get_obj_state: s->obj_tag was set empty 2012-09-11 23:22:48.294879 7f67bd722700 20 get_obj_state: rctx=0x7f6a8800a6f0 obj=.rgw:ocdn state=0x7f6a8800b3b8 s->prefetch_data=0 2012-09-11 23:22:48.294886 7f67bd722700 20 get_obj_state: rctx=0x7f6a8800a6f0 obj=.rgw:ocdn state=0x7f6a8800b3b8 s->prefetch_data=0 2012-09-11 23:22:48.294890 7f67bd722700 20 state for obj=.rgw:ocdn is not atomic, not appending atomic test 2012-09-11 23:22:48.294892 7f67bd722700 20 rados->read obj-ofs=0 read_ofs=0 read_len=16384 2012-09-11 23:22:48.296079 7f67bd722700 20 rados->read r=0 bl.length=65 2012-09-11 23:22:48.296100 7f67bd722700 20 rgw_get_bucket_info: bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 2012-09-11 23:22:48.296112 7f67bd722700 20 get_obj_state: rctx=0x7f6a8800a6f0 obj=ocdn: state=0x7f6a8800b3b8 s->prefetch_data=0 2012-09-11 23:22:48.296129 7f67bd722700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-11 23:22:48.296142 7f67bd722700 2 req 75477:0.008146:s3:PUT /ocdn/test3:put_obj:verifying op permissions 2012-09-11 23:22:48.296146 7f67bd722700 5 Searching permissions for uid=0 mask=2 2012-09-11 23:22:48.296148 7f67bd722700 5 Found permission: 15 2012-09-11 23:22:48.296149 7f67bd722700 10 uid=0 requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 2012-09-11 23:22:48.296151 7f67bd722700 2 req 75477:0.008156:s3:PUT /ocdn/test3:put_obj:verifying op params 2012-09-11 23:22:48.296154 7f67bd722700 2 req 75477:0.008159:s3:PUT /ocdn/test3:put_obj:executing 2012-09-11 23:22:48.296240 7f67bd722700 10 x>> x-amz-acl:public-read 2012-09-11 23:22:48.296246 7f67bd722700 10 x>> x-amz-date:Tue, 11 Sep 2012 21:22:48 GMT 2012-09-11 23:22:48.296258 7f67bd722700 20 get_obj_state: rctx=0x7f6a8800a6f0 obj=ocdn:test3 state=0x7f6a8800d758 s->prefetch_data=0 2012-09-11 23:22:48.297443 7f67bd722700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7f6a8800d758 2012-09-11 23:22:48.315654 7f67bd722700 10 --> ETag: "a241f32b8cf07f90d36b5199 2012-09-11 23:22:48.315678 7f67bd722700 10 --> Content-Length: 0 2012-09-11 23:22:48.315680 7f67bd722700 10 --> Accept-Ranges: bytes 2012-09-11 23:22:48.315682 7f67bd722700 10 --> Status: 200 2012-09-11 23:22:48.315685 7f67bd722700 10 --> Content-type: application/xml 2012-09-11 23:22:48.315801 7f67bd722700 2 req 75477:0.027806:s3:PUT /ocdn/test3:put_obj:http status=200 2012-09-11 23:22:48.316010 7f67bd722700 1 ====== req done req=0x274a630 http_status=200 ====== On Tue, Sep 11, 2012 at 10:46 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: > On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> And more logs: >> >> >> 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done >> req=0x141a650 http_status=403 ====== >> 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 >> 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty >> 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing >> 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> >> 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ >> 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ >> 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ >> 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 >> 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 >> 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 >> 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 >> ...skipping... >> 2012-09-11 22:23:44.530567 7faf0bf4f700 10 >> s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> s->bucket=ocdn >> 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 22:23:44.530595 7faf0bf4f700 20 >> SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530596 7faf0bf4f700 20 >> REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530598 7faf0bf4f700 20 >> DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 >> 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 >> 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 >> 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive >> 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 >> (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like >> Gecko) Chrome/21.0.1180.89 Safari/537.1 >> 2012-09-11 22:23:44.530615 7faf0bf4f700 20 >> HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 >> 2012-09-11 22:23:44.530616 7faf0bf4f700 20 >> HTTP_ACCEPT_ENCODING=gzip,deflate,sdch >> 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 >> 2012-09-11 22:23:44.530618 7faf0bf4f700 20 >> HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 >> 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting >> op >> 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing >> 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading >> permissions >> 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 >> 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 >> obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> state=0x7fb01c00a608 s->prefetch_data=1 >> 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying >> op permissions >> 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=1 >> 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for >> group=1 mask=1 >> 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found >> 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 >> 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=16 >> 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for >> group=1 mask=16 >> 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 >> 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 >> 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 >> 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 >> 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml >> 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http >> status=403 >> 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done >> req=0x13995c0 http_status=403 ====== >> >> Good put, and 403 after: >> >> 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing >> 2012-09-11 22:37:34.677492 7fafddffb700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= >> 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677508 7fafddffb700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677509 7fafddffb700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677510 7fafddffb700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= >> 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.677521 7fafddffb700 20 >> HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue >> 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680452 7fafddffb700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: >> 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681921 7fafddffb700 15 >> auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 >> 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading >> permissions >> 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 >> 2012-09-11 22:37:34.684786 7fafddffb700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op permissions >> 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for >> uid=0 mask=2 >> 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 >> 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 >> 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op params >> 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing >> 2012-09-11 22:37:34.684831 7fafddffb700 15 >> supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 >> 2012-09-11 22:37:34.684841 7fafddffb700 15 >> supplied_md5=ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 >> obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> state=0x7fb00c00bda8 s->prefetch_data=0 >> 2012-09-11 22:37:34.686278 7fafddffb700 20 >> prepare_atomic_for_write_impl: state is not atomic. >> state=0x7fb00c00bda8 >> 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 >> 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 >> 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 >> 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml >> 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >> status=200 >> 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done >> req=0x139a3d0 http_status=200 ====== >> >> >> 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 >> 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty >> 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request >> req=0x1368860 ===== >> 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing >> 2012-09-11 22:37:34.708402 7faec76c6700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read >> 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl >> 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= >> 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708440 7faec76c6700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= >> 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711108 7faec76c6700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: >> 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= >> 2012-09-11 22:37:34.712742 7faec76c6700 15 >> auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 >> 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request > > The request itself fails to authorize. I think it's related to trying > to do canned acl put_acl. Not sure if that's actually supposed to > work. > > Yehuda -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 21:28 ` Sławomir Skowron @ 2012-09-11 21:32 ` Yehuda Sadeh 2012-09-11 21:38 ` Sławomir Skowron 0 siblings, 1 reply; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 21:32 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel On Tue, Sep 11, 2012 at 2:28 PM, Sławomir Skowron <szibis@gmail.com> wrote: > :~# s3 -u put ocdn/test3 cannedAcl=public-read < /tmp/testdl > :~# s3 -u get ocdn/test3 > /tmp/test3 > :~# HEAD http://127.0.0.1/ocdn/test3 > 200 OK > Connection: close > Date: Tue, 11 Sep 2012 21:23:19 GMT > Accept-Ranges: bytes > ETag: "a241f32b8cf07f90d36b5199629b8829" > Server: nginx > Content-Length: 6713 > Last-Modified: Tue, 11 Sep 2012 21:22:48 GMT > Client-Date: Tue, 11 Sep 2012 21:23:19 GMT > Client-Peer: 127.0.0.1:80 > Client-Response-Num: 1 > > put with cannedacl forpublic works on this object. > As it should. I was referring to setacl with canned acl. Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 21:32 ` Yehuda Sadeh @ 2012-09-11 21:38 ` Sławomir Skowron 2012-09-11 21:44 ` Yehuda Sadeh 0 siblings, 1 reply; 16+ messages in thread From: Sławomir Skowron @ 2012-09-11 21:38 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel Ok, but why this happend. There is no new code started before this problem. Is there any way to recover cluster to normal operation withoud Access Denied in s3 any acl operation ?? On Tue, Sep 11, 2012 at 11:32 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: > On Tue, Sep 11, 2012 at 2:28 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> :~# s3 -u put ocdn/test3 cannedAcl=public-read < /tmp/testdl >> :~# s3 -u get ocdn/test3 > /tmp/test3 >> :~# HEAD http://127.0.0.1/ocdn/test3 >> 200 OK >> Connection: close >> Date: Tue, 11 Sep 2012 21:23:19 GMT >> Accept-Ranges: bytes >> ETag: "a241f32b8cf07f90d36b5199629b8829" >> Server: nginx >> Content-Length: 6713 >> Last-Modified: Tue, 11 Sep 2012 21:22:48 GMT >> Client-Date: Tue, 11 Sep 2012 21:23:19 GMT >> Client-Peer: 127.0.0.1:80 >> Client-Response-Num: 1 >> >> put with cannedacl forpublic works on this object. >> > As it should. I was referring to setacl with canned acl. > > Yehuda -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 21:38 ` Sławomir Skowron @ 2012-09-11 21:44 ` Yehuda Sadeh 2012-09-11 21:50 ` Sławomir Skowron 0 siblings, 1 reply; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 21:44 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel Maybe your s3 library got updated and now uses a newer s3 dialect? Basically you need to update the bucket acl, e.g.: # s3 -u create foo # s3 -u getacl foo | s3 -u setacl oldbucket < acl On Tue, Sep 11, 2012 at 2:38 PM, Sławomir Skowron <szibis@gmail.com> wrote: > Ok, but why this happend. There is no new code started before this > problem. Is there any way to recover cluster to normal operation > withoud Access Denied in s3 any acl operation ?? > > On Tue, Sep 11, 2012 at 11:32 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >> On Tue, Sep 11, 2012 at 2:28 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>> :~# s3 -u put ocdn/test3 cannedAcl=public-read < /tmp/testdl >>> :~# s3 -u get ocdn/test3 > /tmp/test3 >>> :~# HEAD http://127.0.0.1/ocdn/test3 >>> 200 OK >>> Connection: close >>> Date: Tue, 11 Sep 2012 21:23:19 GMT >>> Accept-Ranges: bytes >>> ETag: "a241f32b8cf07f90d36b5199629b8829" >>> Server: nginx >>> Content-Length: 6713 >>> Last-Modified: Tue, 11 Sep 2012 21:22:48 GMT >>> Client-Date: Tue, 11 Sep 2012 21:23:19 GMT >>> Client-Peer: 127.0.0.1:80 >>> Client-Response-Num: 1 >>> >>> put with cannedacl forpublic works on this object. >>> >> As it should. I was referring to setacl with canned acl. >> >> Yehuda > > > > -- > ----- > Pozdrawiam > > Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 21:44 ` Yehuda Sadeh @ 2012-09-11 21:50 ` Sławomir Skowron 2012-09-11 21:55 ` Yehuda Sadeh 0 siblings, 1 reply; 16+ messages in thread From: Sławomir Skowron @ 2012-09-11 21:50 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel Ehh my ideas run out for this day. libs3 2.0-1 s3 -u create foo Bucket successfully created. s3 -u getacl foo | s3 -u setacl ocdn < /tmp/acl ERROR: ErrorAccessDenied ERROR: ErrorAccessDenied On Tue, Sep 11, 2012 at 11:44 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: > Maybe your s3 library got updated and now uses a newer s3 dialect? > > Basically you need to update the bucket acl, e.g.: > > # s3 -u create foo > # s3 -u getacl foo | s3 -u setacl oldbucket < acl > > On Tue, Sep 11, 2012 at 2:38 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> Ok, but why this happend. There is no new code started before this >> problem. Is there any way to recover cluster to normal operation >> withoud Access Denied in s3 any acl operation ?? >> >> On Tue, Sep 11, 2012 at 11:32 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>> On Tue, Sep 11, 2012 at 2:28 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>>> :~# s3 -u put ocdn/test3 cannedAcl=public-read < /tmp/testdl >>>> :~# s3 -u get ocdn/test3 > /tmp/test3 >>>> :~# HEAD http://127.0.0.1/ocdn/test3 >>>> 200 OK >>>> Connection: close >>>> Date: Tue, 11 Sep 2012 21:23:19 GMT >>>> Accept-Ranges: bytes >>>> ETag: "a241f32b8cf07f90d36b5199629b8829" >>>> Server: nginx >>>> Content-Length: 6713 >>>> Last-Modified: Tue, 11 Sep 2012 21:22:48 GMT >>>> Client-Date: Tue, 11 Sep 2012 21:23:19 GMT >>>> Client-Peer: 127.0.0.1:80 >>>> Client-Response-Num: 1 >>>> >>>> put with cannedacl forpublic works on this object. >>>> >>> As it should. I was referring to setacl with canned acl. >>> >>> Yehuda >> >> >> >> -- >> ----- >> Pozdrawiam >> >> Sławek "sZiBis" Skowron -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 21:50 ` Sławomir Skowron @ 2012-09-11 21:55 ` Yehuda Sadeh 0 siblings, 0 replies; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 21:55 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel On Tue, Sep 11, 2012 at 2:50 PM, Sławomir Skowron <szibis@gmail.com> wrote: > Ehh my ideas run out for this day. > > libs3 2.0-1 > > s3 -u create foo > > Bucket successfully created. > > s3 -u getacl foo | s3 -u setacl ocdn < /tmp/acl Hmm.. I had a typo # s3 -u getacl foo | s3 -u setacl ocdn -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 20:41 ` Sławomir Skowron 2012-09-11 20:46 ` Yehuda Sadeh @ 2012-09-11 23:27 ` Yehuda Sadeh 2012-09-12 5:07 ` Sławomir Skowron 2012-09-12 14:21 ` [Solved] " Sławomir Skowron 1 sibling, 2 replies; 16+ messages in thread From: Yehuda Sadeh @ 2012-09-11 23:27 UTC (permalink / raw) To: Sławomir Skowron; +Cc: ceph-devel On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: > And more logs: > > > 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done > req=0x141a650 http_status=403 ====== > 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 > 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty > 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request > req=0x139a3d0 ===== > 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing > 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> > 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER > 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= > 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET > 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= > 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= > 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= > 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ > 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ > 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ > 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 > 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 > 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 > 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 > 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 > ...skipping... > 2012-09-11 22:23:44.530567 7faf0bf4f700 10 > s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > s->bucket=ocdn > 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= > 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET > 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= > 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= > 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= > 2012-09-11 22:23:44.530595 7faf0bf4f700 20 > SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530596 7faf0bf4f700 20 > REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530598 7faf0bf4f700 20 > DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 > 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 > 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 > 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 > 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= > 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 > 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 > 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive > 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 > (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like > Gecko) Chrome/21.0.1180.89 Safari/537.1 > 2012-09-11 22:23:44.530615 7faf0bf4f700 20 > HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > 2012-09-11 22:23:44.530616 7faf0bf4f700 20 > HTTP_ACCEPT_ENCODING=gzip,deflate,sdch > 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 > 2012-09-11 22:23:44.530618 7faf0bf4f700 20 > HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 > 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting > op > 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing > 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading > permissions > 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 > s->prefetch_data=0 > 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is > not atomic, not appending atomic test > 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 > 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: > bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 > 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 > 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: > rctx=0x7fb01c0122c0 > obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg > state=0x7fb01c00a608 s->prefetch_data=1 > 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying > op permissions > 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for > uid=anonymous mask=1 > 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found > 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for > group=1 mask=1 > 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found > 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions > id=anonymous owner=0 perm=0 > 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested > perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 > 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for > uid=anonymous mask=16 > 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found > 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for > group=1 mask=16 > 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 > 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions > id=anonymous owner=0 perm=0 > 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested > perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 > 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 > 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 > 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes > 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml > 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET > /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http > status=403 > 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done > req=0x13995c0 http_status=403 ====== > > Good put, and 403 after: > > 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request > req=0x139a3d0 ===== > 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing > 2012-09-11 22:37:34.677492 7fafddffb700 10 > s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > s->bucket=ocdn > 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= > 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT > 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain > 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 > 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 > 2012-09-11 22:37:34.677508 7fafddffb700 20 > SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677509 7fafddffb700 20 > REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677510 7fafddffb700 20 > DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 > 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 > 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 > 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 > 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= > 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 > 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 > 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity > 2012-09-11 22:37:34.677521 7fafddffb700 20 > HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== > 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue > 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 > 20:37:46 GMT > 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) > 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain > 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS > 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting > op > 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing > 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb00c004a18 s->prefetch_data=0 > 2012-09-11 22:37:34.680452 7fafddffb700 20 state for > obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic > test > 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 > 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): > dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: missing some lines here, did you grep the output? > 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.681921 7fafddffb700 15 > auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= > 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 > 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading > permissions > 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 > s->prefetch_data=0 > 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is > not atomic, not appending atomic test > 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 > 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: > bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 > 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 > 2012-09-11 22:37:34.684786 7fafddffb700 15 Read > AccessControlPolicy<AccessControlPolicy > xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> > 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying > op permissions > 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for > uid=0 mask=2 > 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 > 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm > (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 > 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying > op params > 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing > 2012-09-11 22:37:34.684831 7fafddffb700 15 > supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== > 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 > 2012-09-11 22:37:34.684841 7fafddffb700 15 > supplied_md5=ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: > rctx=0x7fb00c007050 > obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > state=0x7fb00c00bda8 s->prefetch_data=0 > 2012-09-11 22:37:34.686278 7fafddffb700 20 > prepare_atomic_for_write_impl: state is not atomic. > state=0x7fb00c00bda8 > 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 > 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 > 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes > 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 > 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml > 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http > status=200 > 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done > req=0x139a3d0 http_status=200 ====== > > > 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 > 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty > 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request > req=0x1368860 ===== > 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing > 2012-09-11 22:37:34.708402 7faec76c6700 10 > s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > s->bucket=ocdn > 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read > 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read > 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER > 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi > 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl > 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT > 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= > 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 > 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 > 2012-09-11 22:37:34.708440 7faec76c6700 20 > SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708441 7faec76c6700 20 > REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708441 7faec76c6700 20 > DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www > 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 > 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 > 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 > 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 > 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 > 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 > 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 > 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= > 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 > 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no > 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 > 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity > 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 > 20:37:46 GMT > 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read > 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS > 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= > 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) > 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting > op > 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing > 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag > was set empty > 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: > rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A > state=0x7fb2a4007f18 s->prefetch_data=0 > 2012-09-11 22:37:34.711108 7faec76c6700 20 state for > obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic > test > 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 > read_ofs=0 read_len=16384 > 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 > 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): > dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 > 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: and here also > 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= > 2012-09-11 22:37:34.712742 7faec76c6700 15 > auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= > 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 > 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request > 2012-09-11 22:37:34.712754 7faec76c6700 10 --> Status: 403 > 2012-09-11 22:37:34.712766 7faec76c6700 10 --> Content-Length: 78 > 2012-09-11 22:37:34.712769 7faec76c6700 10 --> Accept-Ranges: bytes > 2012-09-11 22:37:34.712772 7faec76c6700 10 --> Content-type: application/xml > 2012-09-11 22:37:34.712887 7faec76c6700 2 req 71188:0.004554:s3:PUT > /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http > status=403 > 2012-09-11 22:37:34.713093 7faec76c6700 1 ====== req done > req=0x1368860 http_status=403 ====== > > On Tue, Sep 11, 2012 at 7:46 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>>>> Every acl operation ending with 403 in PUT. >>>>> >>>>> ~# s3 -u test oc >>>>> Bucket Status >>>>> -------------------------------------------------------- -------------------- >>>>> oc Access Denied >>>>> >>>>> Anyone know why, and how to enable this bucket ?? Now i have problems >>>>> with cluster, because there is no way to upload new file >>>>> >>>>> ~# s3 -u getacl oc >>>>> >>>>> ERROR: ErrorAccessDenied >>>>> >>>> >>>> User somehow lost bucket ownership (was it actually the owner?). Do >>>> you know how to reproduce the issue? any remaining logs? >>>> >>>> Try getting bucket info: >>>> >>>> # radosgw-admin bucket stats --bucket=oc >>>> >>>> If that doesn't fail and actually shows relevant info, try checking >>>> whether the user credentials match the s3 tool credentials. >>>> >>> Oh, and thinking about it some more.. 'oc' is a too short name for a >>> bucket (requires min of 3 chars). How did you create it? The failure >>> may be related. >> >> Yes i made a shortcut of name :)) >> >> Right now every bucket in pool, are afected >> >> :~#radosgw-admin bucket stats --bucket=lvstest >> { "bucket": "lvstest", >> "pool": ".rgw.buckets", >> "id": "1142048.1", >> "marker": "1142048.1", >> "owner": "0", >> "usage": { "rgw.main": { "size_kb": 1, >> "size_kb_actual": 4, >> "num_objects": 1}}} >> :~# radosgw-admin bucket stats --bucket=ocdn >> { "bucket": "ocdn", >> "pool": ".rgw.buckets", >> "id": "4168.2", >> "marker": "4168.2", >> "owner": "0", >> "usage": { "rgw.main": { "size_kb": 513059717, >> "size_kb_actual": 516402364, >> "num_objects": 1606730}}} >> >> Credentials from radosgw-admin user info match that from clients requests. >> >> Every GET, PUT, HEAD using this credentials works fine, but only one >> operations does not work (403 from radosgw) - setting acl for object >> for a public-read. Setting canned acl with PUT for public-read from >> s3lib work good, but get/set acl failed. >> >> list bucket object works good, and list buckets via s3 client. >> >> Now i can't reproduce, but i will dig logs from radosgw, for related >> time, when this happend. >> >> Example 403 from radosgw log, before that PUT of object ends with 200: >> >> 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done >> req=0x1435980 http_status=403 ====== >> 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 >> 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty >> 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request >> req=0x13994c0 ===== >> 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing >> 2012-09-11 19:37:04.342971 7fb25d7fa700 10 >> s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >> s->bucket=ocdn >> 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read >> 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read >> 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl >> 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT >> 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= >> 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 >> 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 >> 2012-09-11 19:37:04.343005 7fb25d7fa700 20 >> SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >> 2012-09-11 19:37:04.343006 7fb25d7fa700 20 >> REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >> 2012-09-11 19:37:04.343007 7fb25d7fa700 20 >> DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >> 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 >> 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 >> 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 >> 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= >> 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 >> 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no >> 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 17:37:02 GMT >> 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read >> 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= >> 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) >> 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT >> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting >> op >> 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT >> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing >> 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: >> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb23c00b468 s->prefetch_data=0 >> 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: >> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb23c00b468 s->prefetch_data=0 >> 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: >> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb23c00b468 s->prefetch_data=0 >> 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 >> 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): >> dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >> 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: >> 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= >> 2012-09-11 19:37:04.347198 7fb25d7fa700 15 >> auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= >> 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 >> 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request >> 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 >> 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 >> 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes >> 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml >> 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT >> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http >> status=403 >> 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done >> req=0x13994c0 http_status=403 ====== >> 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 >> >> Interesting is that second cluster, which is replicated via s3 client, >> get the same issue, from that in production one. Is this can be >> generated by many error on radosgw layer ?? >> >> I will try to find beginning of this problems in logs. >> >>> >>> Yehuda >> >> >> >> -- >> ----- >> Pozdrawiam >> >> Sławek "sZiBis" Skowron > > > > -- > ----- > Pozdrawiam > > Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-11 23:27 ` Yehuda Sadeh @ 2012-09-12 5:07 ` Sławomir Skowron 2012-09-12 9:14 ` Sławomir Skowron 2012-09-12 14:21 ` [Solved] " Sławomir Skowron 1 sibling, 1 reply; 16+ messages in thread From: Sławomir Skowron @ 2012-09-12 5:07 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel@vger.kernel.org I grep only for 7faec76c6700 Where are stored acl data for bucket in ceph?? Maybe acl are broken for anonymous user ?? Ceph supporting global acl for bucket?? Dnia 12 wrz 2012 o godz. 01:27 Yehuda Sadeh <yehuda@inktank.com> napisał(a): > On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> And more logs: >> >> >> 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done >> req=0x141a650 http_status=403 ====== >> 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 >> 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty >> 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing >> 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> >> 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ >> 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ >> 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ >> 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 >> 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 >> 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 >> 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 >> ...skipping... >> 2012-09-11 22:23:44.530567 7faf0bf4f700 10 >> s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> s->bucket=ocdn >> 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 22:23:44.530595 7faf0bf4f700 20 >> SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530596 7faf0bf4f700 20 >> REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530598 7faf0bf4f700 20 >> DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 >> 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 >> 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 >> 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive >> 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 >> (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like >> Gecko) Chrome/21.0.1180.89 Safari/537.1 >> 2012-09-11 22:23:44.530615 7faf0bf4f700 20 >> HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 >> 2012-09-11 22:23:44.530616 7faf0bf4f700 20 >> HTTP_ACCEPT_ENCODING=gzip,deflate,sdch >> 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 >> 2012-09-11 22:23:44.530618 7faf0bf4f700 20 >> HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 >> 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting >> op >> 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing >> 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading >> permissions >> 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 >> 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 >> obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> state=0x7fb01c00a608 s->prefetch_data=1 >> 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying >> op permissions >> 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=1 >> 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for >> group=1 mask=1 >> 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found >> 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 >> 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=16 >> 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for >> group=1 mask=16 >> 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 >> 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 >> 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 >> 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 >> 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml >> 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http >> status=403 >> 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done >> req=0x13995c0 http_status=403 ====== >> >> Good put, and 403 after: >> >> 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing >> 2012-09-11 22:37:34.677492 7fafddffb700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= >> 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677508 7fafddffb700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677509 7fafddffb700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677510 7fafddffb700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= >> 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.677521 7fafddffb700 20 >> HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue >> 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680452 7fafddffb700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: > > missing some lines here, did you grep the output? > >> 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681921 7fafddffb700 15 >> auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 >> 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading >> permissions >> 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 >> 2012-09-11 22:37:34.684786 7fafddffb700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op permissions >> 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for >> uid=0 mask=2 >> 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 >> 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 >> 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op params >> 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing >> 2012-09-11 22:37:34.684831 7fafddffb700 15 >> supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 >> 2012-09-11 22:37:34.684841 7fafddffb700 15 >> supplied_md5=ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 >> obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> state=0x7fb00c00bda8 s->prefetch_data=0 >> 2012-09-11 22:37:34.686278 7fafddffb700 20 >> prepare_atomic_for_write_impl: state is not atomic. >> state=0x7fb00c00bda8 >> 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 >> 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 >> 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 >> 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml >> 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >> status=200 >> 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done >> req=0x139a3d0 http_status=200 ====== >> >> >> 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 >> 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty >> 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request >> req=0x1368860 ===== >> 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing >> 2012-09-11 22:37:34.708402 7faec76c6700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read >> 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl >> 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= >> 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708440 7faec76c6700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= >> 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711108 7faec76c6700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: > > and here also > >> 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= >> 2012-09-11 22:37:34.712742 7faec76c6700 15 >> auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 >> 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request >> 2012-09-11 22:37:34.712754 7faec76c6700 10 --> Status: 403 >> 2012-09-11 22:37:34.712766 7faec76c6700 10 --> Content-Length: 78 >> 2012-09-11 22:37:34.712769 7faec76c6700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:37:34.712772 7faec76c6700 10 --> Content-type: application/xml >> 2012-09-11 22:37:34.712887 7faec76c6700 2 req 71188:0.004554:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >> status=403 >> 2012-09-11 22:37:34.713093 7faec76c6700 1 ====== req done >> req=0x1368860 http_status=403 ====== >> >> On Tue, Sep 11, 2012 at 7:46 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>> On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>>>>> Every acl operation ending with 403 in PUT. >>>>>> >>>>>> ~# s3 -u test oc >>>>>> Bucket Status >>>>>> -------------------------------------------------------- -------------------- >>>>>> oc Access Denied >>>>>> >>>>>> Anyone know why, and how to enable this bucket ?? Now i have problems >>>>>> with cluster, because there is no way to upload new file >>>>>> >>>>>> ~# s3 -u getacl oc >>>>>> >>>>>> ERROR: ErrorAccessDenied >>>>>> >>>>> >>>>> User somehow lost bucket ownership (was it actually the owner?). Do >>>>> you know how to reproduce the issue? any remaining logs? >>>>> >>>>> Try getting bucket info: >>>>> >>>>> # radosgw-admin bucket stats --bucket=oc >>>>> >>>>> If that doesn't fail and actually shows relevant info, try checking >>>>> whether the user credentials match the s3 tool credentials. >>>>> >>>> Oh, and thinking about it some more.. 'oc' is a too short name for a >>>> bucket (requires min of 3 chars). How did you create it? The failure >>>> may be related. >>> >>> Yes i made a shortcut of name :)) >>> >>> Right now every bucket in pool, are afected >>> >>> :~#radosgw-admin bucket stats --bucket=lvstest >>> { "bucket": "lvstest", >>> "pool": ".rgw.buckets", >>> "id": "1142048.1", >>> "marker": "1142048.1", >>> "owner": "0", >>> "usage": { "rgw.main": { "size_kb": 1, >>> "size_kb_actual": 4, >>> "num_objects": 1}}} >>> :~# radosgw-admin bucket stats --bucket=ocdn >>> { "bucket": "ocdn", >>> "pool": ".rgw.buckets", >>> "id": "4168.2", >>> "marker": "4168.2", >>> "owner": "0", >>> "usage": { "rgw.main": { "size_kb": 513059717, >>> "size_kb_actual": 516402364, >>> "num_objects": 1606730}}} >>> >>> Credentials from radosgw-admin user info match that from clients requests. >>> >>> Every GET, PUT, HEAD using this credentials works fine, but only one >>> operations does not work (403 from radosgw) - setting acl for object >>> for a public-read. Setting canned acl with PUT for public-read from >>> s3lib work good, but get/set acl failed. >>> >>> list bucket object works good, and list buckets via s3 client. >>> >>> Now i can't reproduce, but i will dig logs from radosgw, for related >>> time, when this happend. >>> >>> Example 403 from radosgw log, before that PUT of object ends with 200: >>> >>> 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done >>> req=0x1435980 http_status=403 ====== >>> 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 >>> 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty >>> 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request >>> req=0x13994c0 ===== >>> 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing >>> 2012-09-11 19:37:04.342971 7fb25d7fa700 10 >>> s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> s->bucket=ocdn >>> 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read >>> 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl >>> 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT >>> 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= >>> 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 >>> 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 >>> 2012-09-11 19:37:04.343005 7fb25d7fa700 20 >>> SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343006 7fb25d7fa700 20 >>> REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343007 7fb25d7fa700 20 >>> DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 >>> 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 >>> 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 >>> 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 >>> 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 >>> 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= >>> 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 >>> 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no >>> 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 >>> 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity >>> 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 >>> 17:37:02 GMT >>> 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS >>> 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= >>> 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) >>> 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting >>> op >>> 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing >>> 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for >>> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >>> test >>> 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 >>> 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): >>> dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: >>> 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= >>> 2012-09-11 19:37:04.347198 7fb25d7fa700 15 >>> auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= >>> 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 >>> 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request >>> 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 >>> 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 >>> 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes >>> 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml >>> 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http >>> status=403 >>> 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done >>> req=0x13994c0 http_status=403 ====== >>> 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 >>> >>> Interesting is that second cluster, which is replicated via s3 client, >>> get the same issue, from that in production one. Is this can be >>> generated by many error on radosgw layer ?? >>> >>> I will try to find beginning of this problems in logs. >>> >>>> >>>> Yehuda >>> >>> >>> >>> -- >>> ----- >>> Pozdrawiam >>> >>> Sławek "sZiBis" Skowron >> >> >> >> -- >> ----- >> Pozdrawiam >> >> Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Access Dienied for bucket upload - 403 code 2012-09-12 5:07 ` Sławomir Skowron @ 2012-09-12 9:14 ` Sławomir Skowron 0 siblings, 0 replies; 16+ messages in thread From: Sławomir Skowron @ 2012-09-12 9:14 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel@vger.kernel.org Same file, same s3 credentials on two clusters. Really strange, good: 2012-09-12 09:30:51.935383 7f0939ffb700 5 Searching permissions for uid=anonymous mask=1 2012-09-12 09:30:51.935385 7f0939ffb700 5 Permissions for user not found 2012-09-12 09:30:51.935387 7f0939ffb700 5 Searching permissions for group=1 mask=1 2012-09-12 09:30:51.935388 7f0939ffb700 5 Found permission: 1 2012-09-12 09:30:51.935389 7f0939ffb700 5 Getting permissions id=anonymous owner=0 perm=1 2012-09-12 09:30:51.935390 7f0939ffb700 10 uid=anonymous requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 bad: 2012-09-12 09:41:21.440991 7fa257fff700 5 Searching permissions for uid=anonymous mask=1 2012-09-12 09:41:21.440994 7fa257fff700 5 Permissions for user not found 2012-09-12 09:41:21.440995 7fa257fff700 5 Searching permissions for group=1 mask=1 2012-09-12 09:41:21.440996 7fa257fff700 5 Permissions for group not found 2012-09-12 09:41:21.440997 7fa257fff700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-12 09:41:21.440999 7fa257fff700 10 uid=anonymous requested perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 2012-09-12 09:41:21.441001 7fa257fff700 5 Searching permissions for uid=anonymous mask=16 2012-09-12 09:41:21.441003 7fa257fff700 5 Permissions for user not found 2012-09-12 09:41:21.441004 7fa257fff700 5 Searching permissions for group=1 mask=16 2012-09-12 09:41:21.441005 7fa257fff700 5 Found permission: 1 2012-09-12 09:41:21.441006 7fa257fff700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-12 09:41:21.441007 7fa257fff700 10 uid=anonymous requested perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 What this mean ?? Why access policy in read are diffrent ?? Full logs. Second cluster GET works ok: 2012-09-12 09:30:51.933114 7f0939ffb700 20 rgw_get_bucket_info: bucket=ocdn(@.rgw.buckets[8700.1]) owner 0 2012-09-12 09:30:51.933126 7f0939ffb700 20 get_obj_state: rctx=0x7f091400c1f0 obj=ocdn: state=0x7f0914022288 s->prefetch_data=0 2012-09-12 09:30:51.933144 7f0939ffb700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-12 09:30:51.933165 7f0939ffb700 20 get_obj_state: rctx=0x7f091400c1f0 obj=ocdn:images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg state=0x7f0914022c48 s->prefetch_data=1 2012-09-12 09:30:51.935344 7f0939ffb700 20 get_obj_state: s->obj_tag was set empty 2012-09-12 09:30:51.935364 7f0939ffb700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-12 09:30:51.935376 7f0939ffb700 2 req 16954:0.005555:s3:GET /ocdn/images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg:get_obj:verifying op permissions 2012-09-12 09:30:51.935383 7f0939ffb700 5 Searching permissions for uid=anonymous mask=1 2012-09-12 09:30:51.935385 7f0939ffb700 5 Permissions for user not found 2012-09-12 09:30:51.935387 7f0939ffb700 5 Searching permissions for group=1 mask=1 2012-09-12 09:30:51.935388 7f0939ffb700 5 Found permission: 1 2012-09-12 09:30:51.935389 7f0939ffb700 5 Getting permissions id=anonymous owner=0 perm=1 2012-09-12 09:30:51.935390 7f0939ffb700 10 uid=anonymous requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2012-09-12 09:30:51.935393 7f0939ffb700 2 req 16954:0.005573:s3:GET /ocdn/images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg:get_obj:verifying op params 2012-09-12 09:30:51.935395 7f0939ffb700 2 req 16954:0.005575:s3:GET /ocdn/images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg:get_obj:executing 2012-09-12 09:30:51.935406 7f0939ffb700 20 get_obj_state: rctx=0x7f091400c1f0 obj=ocdn:images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg state=0x7f0914022c48 s->prefetch_data=1 2012-09-12 09:30:51.935413 7f0939ffb700 20 Read xattr: user.rgw.acl 2012-09-12 09:30:51.935414 7f0939ffb700 20 Read xattr: user.rgw.content_type 2012-09-12 09:30:51.935415 7f0939ffb700 20 Read xattr: user.rgw.etag 2012-09-12 09:30:51.935416 7f0939ffb700 20 Read xattr: user.rgw.idtag 2012-09-12 09:30:51.935417 7f0939ffb700 20 Read xattr: user.rgw.manifest 2012-09-12 09:30:51.935418 7f0939ffb700 20 Read xattr: user.rgw.x-amz-acl 2012-09-12 09:30:51.935418 7f0939ffb700 20 Read xattr: user.rgw.x-amz-date 2012-09-12 09:30:51.935422 7f0939ffb700 20 get_obj_state: rctx=0x7f091400c1f0 obj=ocdn:images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg state=0x7f0914022c48 s->prefetch_data=1 2012-09-12 09:30:51.935424 7f0939ffb700 20 get_obj_state: rctx=0x7f091400c1f0 obj=ocdn:images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg state=0x7f0914022c48 s->prefetch_data=1 2012-09-12 09:30:51.935426 7f0939ffb700 20 state->obj_tag is empty, not appending atomic test 2012-09-12 09:30:51.935436 7f0939ffb700 10 --> Content-Length: 2219 2012-09-12 09:30:51.935438 7f0939ffb700 10 --> Accept-Ranges: bytes 2012-09-12 09:30:51.935447 7f0939ffb700 10 --> Last-Modified: Tue, 11 Sep 2012 2012-09-12 09:30:51.935451 7f0939ffb700 10 --> ETag: "cd96f6c7632c9f782967c665 2012-09-12 09:30:51.935456 7f0939ffb700 10 --> Status: 200 2012-09-12 09:30:51.935458 7f0939ffb700 10 --> Content-type: 2012-09-12 09:30:51.935574 7f0939ffb700 2 req 16954:0.005754:s3:GET /ocdn/images/pulscms/YTU7MDYsNWEsM2E_/d4716c54c40099f7df31224486020516.jpg:get_obj:http status=200 first cluster with problem: 2012-09-12 09:41:21.438096 7fa257fff700 20 rgw_get_bucket_info: bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 2012-09-12 09:41:21.438110 7fa257fff700 20 get_obj_state: rctx=0x7fa11c0137b0 obj=ocdn: state=0x7fa11c00ee18 s->prefetch_data=0 2012-09-12 09:41:21.438126 7fa257fff700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-12 09:41:21.438148 7fa257fff700 20 get_obj_state: rctx=0x7fa11c0137b0 obj=ocdn:images/pulscms/ZWI7MDMsMWUwLDAsMSwx/2cbff537b4543942d6571124b9cc3910.jpg state=0x7fa11c0122f8 s->prefetch_data=1 2012-09-12 09:41:21.440954 7fa257fff700 20 get_obj_state: s->obj_tag was set empty 2012-09-12 09:41:21.440972 7fa257fff700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2012-09-12 09:41:21.440984 7fa257fff700 2 req 7430:0.006465:s3:GET /ocdn/images/pulscms/ZWI7MDMsMWUwLDAsMSwx/2cbff537b4543942d6571124b9cc3910.jpg:get_obj:verifying op permissions 2012-09-12 09:41:21.440991 7fa257fff700 5 Searching permissions for uid=anonymous mask=1 2012-09-12 09:41:21.440994 7fa257fff700 5 Permissions for user not found 2012-09-12 09:41:21.440995 7fa257fff700 5 Searching permissions for group=1 mask=1 2012-09-12 09:41:21.440996 7fa257fff700 5 Permissions for group not found 2012-09-12 09:41:21.440997 7fa257fff700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-12 09:41:21.440999 7fa257fff700 10 uid=anonymous requested perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 2012-09-12 09:41:21.441001 7fa257fff700 5 Searching permissions for uid=anonymous mask=16 2012-09-12 09:41:21.441003 7fa257fff700 5 Permissions for user not found 2012-09-12 09:41:21.441004 7fa257fff700 5 Searching permissions for group=1 mask=16 2012-09-12 09:41:21.441005 7fa257fff700 5 Found permission: 1 2012-09-12 09:41:21.441006 7fa257fff700 5 Getting permissions id=anonymous owner=0 perm=0 2012-09-12 09:41:21.441007 7fa257fff700 10 uid=anonymous requested perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 2012-09-12 09:41:21.441016 7fa257fff700 10 --> Status: 403 2012-09-12 09:41:21.441027 7fa257fff700 10 --> Content-Length: 78 2012-09-12 09:41:21.441029 7fa257fff700 10 --> Accept-Ranges: bytes 2012-09-12 09:41:21.441032 7fa257fff700 10 --> Content-type: application/xml 2012-09-12 09:41:21.441138 7fa257fff700 2 req 7430:0.006620:s3:GET /ocdn/images/pulscms/ZWI7MDMsMWUwLDAsMSwx/2cbff537b4543942d6571124b9cc3910.jpg:get_obj:http status=403 On Wed, Sep 12, 2012 at 7:07 AM, Sławomir Skowron <szibis@gmail.com> wrote: > I grep only for 7faec76c6700 > > Where are stored acl data for bucket in ceph?? Maybe acl are broken > for anonymous user ?? > > Ceph supporting global acl for bucket?? > > Dnia 12 wrz 2012 o godz. 01:27 Yehuda Sadeh <yehuda@inktank.com> napisał(a): > >> On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>> And more logs: >>> >>> >>> 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done >>> req=0x141a650 http_status=403 ====== >>> 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 >>> 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty >>> 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request >>> req=0x139a3d0 ===== >>> 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing >>> 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> >>> 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= >>> 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET >>> 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= >>> 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= >>> 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >>> 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ >>> 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ >>> 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ >>> 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 >>> 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 >>> 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 >>> 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >>> 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 >>> ...skipping... >>> 2012-09-11 22:23:44.530567 7faf0bf4f700 10 >>> s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >>> s->bucket=ocdn >>> 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= >>> 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET >>> 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= >>> 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= >>> 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >>> 2012-09-11 22:23:44.530595 7faf0bf4f700 20 >>> SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >>> 2012-09-11 22:23:44.530596 7faf0bf4f700 20 >>> REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >>> 2012-09-11 22:23:44.530598 7faf0bf4f700 20 >>> DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >>> 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 >>> 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 >>> 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 >>> 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >>> 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 >>> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= >>> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 >>> 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no >>> 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 >>> 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive >>> 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 >>> (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like >>> Gecko) Chrome/21.0.1180.89 Safari/537.1 >>> 2012-09-11 22:23:44.530615 7faf0bf4f700 20 >>> HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 >>> 2012-09-11 22:23:44.530616 7faf0bf4f700 20 >>> HTTP_ACCEPT_ENCODING=gzip,deflate,sdch >>> 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 >>> 2012-09-11 22:23:44.530618 7faf0bf4f700 20 >>> HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 >>> 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET >>> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting >>> op >>> 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET >>> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing >>> 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET >>> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading >>> permissions >>> 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: >>> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >>> s->prefetch_data=0 >>> 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: >>> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >>> s->prefetch_data=0 >>> 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: >>> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >>> s->prefetch_data=0 >>> 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is >>> not atomic, not appending atomic test >>> 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 >>> 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: >>> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >>> 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: >>> rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 >>> 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read >>> AccessControlPolicy<AccessControlPolicy >>> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >>> 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: >>> rctx=0x7fb01c0122c0 >>> obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >>> state=0x7fb01c00a608 s->prefetch_data=1 >>> 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read >>> AccessControlPolicy<AccessControlPolicy >>> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >>> 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET >>> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying >>> op permissions >>> 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for >>> uid=anonymous mask=1 >>> 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found >>> 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for >>> group=1 mask=1 >>> 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found >>> 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions >>> id=anonymous owner=0 perm=0 >>> 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested >>> perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 >>> 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for >>> uid=anonymous mask=16 >>> 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found >>> 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for >>> group=1 mask=16 >>> 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 >>> 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions >>> id=anonymous owner=0 perm=0 >>> 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested >>> perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 >>> 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 >>> 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 >>> 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes >>> 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml >>> 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET >>> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http >>> status=403 >>> 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done >>> req=0x13995c0 http_status=403 ====== >>> >>> Good put, and 403 after: >>> >>> 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request >>> req=0x139a3d0 ===== >>> 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing >>> 2012-09-11 22:37:34.677492 7fafddffb700 10 >>> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> s->bucket=ocdn >>> 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= >>> 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT >>> 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain >>> 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 >>> 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 >>> 2012-09-11 22:37:34.677508 7fafddffb700 20 >>> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.677509 7fafddffb700 20 >>> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.677510 7fafddffb700 20 >>> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 >>> 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 >>> 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 >>> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 >>> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 >>> 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= >>> 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 >>> 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no >>> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 >>> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity >>> 2012-09-11 22:37:34.677521 7fafddffb700 20 >>> HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== >>> 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue >>> 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 >>> 20:37:46 GMT >>> 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >>> 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain >>> 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS >>> 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= >>> 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >>> op >>> 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >>> 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb00c004a18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb00c004a18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb00c004a18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.680452 7fafddffb700 20 state for >>> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >>> test >>> 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 >>> 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): >>> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: >> >> missing some lines here, did you grep the output? >> >>> 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= >>> 2012-09-11 22:37:34.681921 7fafddffb700 15 >>> auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= >>> 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 >>> 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading >>> permissions >>> 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >>> s->prefetch_data=0 >>> 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >>> s->prefetch_data=0 >>> 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >>> s->prefetch_data=0 >>> 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is >>> not atomic, not appending atomic test >>> 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 >>> 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: >>> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >>> 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 >>> 2012-09-11 22:37:34.684786 7fafddffb700 15 Read >>> AccessControlPolicy<AccessControlPolicy >>> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >>> 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >>> op permissions >>> 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for >>> uid=0 mask=2 >>> 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 >>> 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm >>> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 >>> 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >>> op params >>> 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing >>> 2012-09-11 22:37:34.684831 7fafddffb700 15 >>> supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== >>> 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 >>> 2012-09-11 22:37:34.684841 7fafddffb700 15 >>> supplied_md5=ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: >>> rctx=0x7fb00c007050 >>> obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> state=0x7fb00c00bda8 s->prefetch_data=0 >>> 2012-09-11 22:37:34.686278 7fafddffb700 20 >>> prepare_atomic_for_write_impl: state is not atomic. >>> state=0x7fb00c00bda8 >>> 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 >>> 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 >>> 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes >>> 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 >>> 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml >>> 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >>> status=200 >>> 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done >>> req=0x139a3d0 http_status=200 ====== >>> >>> >>> 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 >>> 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty >>> 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request >>> req=0x1368860 ===== >>> 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing >>> 2012-09-11 22:37:34.708402 7faec76c6700 10 >>> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> s->bucket=ocdn >>> 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read >>> 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl >>> 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT >>> 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= >>> 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 >>> 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 >>> 2012-09-11 22:37:34.708440 7faec76c6700 20 >>> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.708441 7faec76c6700 20 >>> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.708441 7faec76c6700 20 >>> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 >>> 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 >>> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 >>> 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 >>> 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 >>> 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= >>> 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 >>> 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no >>> 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 >>> 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity >>> 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 >>> 20:37:46 GMT >>> 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS >>> 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= >>> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >>> 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >>> op >>> 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >>> 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: >>> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb2a4007f18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: >>> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb2a4007f18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: >>> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb2a4007f18 s->prefetch_data=0 >>> 2012-09-11 22:37:34.711108 7faec76c6700 20 state for >>> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >>> test >>> 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 >>> 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): >>> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >>> 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: >> >> and here also >> >>> 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= >>> 2012-09-11 22:37:34.712742 7faec76c6700 15 >>> auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= >>> 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 >>> 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request >>> 2012-09-11 22:37:34.712754 7faec76c6700 10 --> Status: 403 >>> 2012-09-11 22:37:34.712766 7faec76c6700 10 --> Content-Length: 78 >>> 2012-09-11 22:37:34.712769 7faec76c6700 10 --> Accept-Ranges: bytes >>> 2012-09-11 22:37:34.712772 7faec76c6700 10 --> Content-type: application/xml >>> 2012-09-11 22:37:34.712887 7faec76c6700 2 req 71188:0.004554:s3:PUT >>> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >>> status=403 >>> 2012-09-11 22:37:34.713093 7faec76c6700 1 ====== req done >>> req=0x1368860 http_status=403 ====== >>> >>> On Tue, Sep 11, 2012 at 7:46 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>>> On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>>> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>>>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>>>>>> Every acl operation ending with 403 in PUT. >>>>>>> >>>>>>> ~# s3 -u test oc >>>>>>> Bucket Status >>>>>>> -------------------------------------------------------- -------------------- >>>>>>> oc Access Denied >>>>>>> >>>>>>> Anyone know why, and how to enable this bucket ?? Now i have problems >>>>>>> with cluster, because there is no way to upload new file >>>>>>> >>>>>>> ~# s3 -u getacl oc >>>>>>> >>>>>>> ERROR: ErrorAccessDenied >>>>>>> >>>>>> >>>>>> User somehow lost bucket ownership (was it actually the owner?). Do >>>>>> you know how to reproduce the issue? any remaining logs? >>>>>> >>>>>> Try getting bucket info: >>>>>> >>>>>> # radosgw-admin bucket stats --bucket=oc >>>>>> >>>>>> If that doesn't fail and actually shows relevant info, try checking >>>>>> whether the user credentials match the s3 tool credentials. >>>>>> >>>>> Oh, and thinking about it some more.. 'oc' is a too short name for a >>>>> bucket (requires min of 3 chars). How did you create it? The failure >>>>> may be related. >>>> >>>> Yes i made a shortcut of name :)) >>>> >>>> Right now every bucket in pool, are afected >>>> >>>> :~#radosgw-admin bucket stats --bucket=lvstest >>>> { "bucket": "lvstest", >>>> "pool": ".rgw.buckets", >>>> "id": "1142048.1", >>>> "marker": "1142048.1", >>>> "owner": "0", >>>> "usage": { "rgw.main": { "size_kb": 1, >>>> "size_kb_actual": 4, >>>> "num_objects": 1}}} >>>> :~# radosgw-admin bucket stats --bucket=ocdn >>>> { "bucket": "ocdn", >>>> "pool": ".rgw.buckets", >>>> "id": "4168.2", >>>> "marker": "4168.2", >>>> "owner": "0", >>>> "usage": { "rgw.main": { "size_kb": 513059717, >>>> "size_kb_actual": 516402364, >>>> "num_objects": 1606730}}} >>>> >>>> Credentials from radosgw-admin user info match that from clients requests. >>>> >>>> Every GET, PUT, HEAD using this credentials works fine, but only one >>>> operations does not work (403 from radosgw) - setting acl for object >>>> for a public-read. Setting canned acl with PUT for public-read from >>>> s3lib work good, but get/set acl failed. >>>> >>>> list bucket object works good, and list buckets via s3 client. >>>> >>>> Now i can't reproduce, but i will dig logs from radosgw, for related >>>> time, when this happend. >>>> >>>> Example 403 from radosgw log, before that PUT of object ends with 200: >>>> >>>> 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done >>>> req=0x1435980 http_status=403 ====== >>>> 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 >>>> 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty >>>> 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request >>>> req=0x13994c0 ===== >>>> 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing >>>> 2012-09-11 19:37:04.342971 7fb25d7fa700 10 >>>> s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>>> s->bucket=ocdn >>>> 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read >>>> 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read >>>> 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER >>>> 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>>> 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl >>>> 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT >>>> 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= >>>> 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 >>>> 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 >>>> 2012-09-11 19:37:04.343005 7fb25d7fa700 20 >>>> SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>>> 2012-09-11 19:37:04.343006 7fb25d7fa700 20 >>>> REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>>> 2012-09-11 19:37:04.343007 7fb25d7fa700 20 >>>> DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>>> 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www >>>> 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 >>>> 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 >>>> 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 >>>> 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 >>>> 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 >>>> 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 >>>> 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 >>>> 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= >>>> 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 >>>> 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no >>>> 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 >>>> 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity >>>> 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 >>>> 17:37:02 GMT >>>> 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read >>>> 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS >>>> 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= >>>> 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) >>>> 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT >>>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting >>>> op >>>> 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT >>>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing >>>> 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: >>>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>>> state=0x7fb23c00b468 s->prefetch_data=0 >>>> 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag >>>> was set empty >>>> 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: >>>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>>> state=0x7fb23c00b468 s->prefetch_data=0 >>>> 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag >>>> was set empty >>>> 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: >>>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>>> state=0x7fb23c00b468 s->prefetch_data=0 >>>> 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for >>>> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >>>> test >>>> 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 >>>> read_ofs=0 read_len=16384 >>>> 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 >>>> 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): >>>> dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>>> 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: >>>> 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= >>>> 2012-09-11 19:37:04.347198 7fb25d7fa700 15 >>>> auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= >>>> 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 >>>> 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request >>>> 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 >>>> 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 >>>> 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes >>>> 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml >>>> 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT >>>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http >>>> status=403 >>>> 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done >>>> req=0x13994c0 http_status=403 ====== >>>> 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 >>>> >>>> Interesting is that second cluster, which is replicated via s3 client, >>>> get the same issue, from that in production one. Is this can be >>>> generated by many error on radosgw layer ?? >>>> >>>> I will try to find beginning of this problems in logs. >>>> >>>>> >>>>> Yehuda >>>> >>>> >>>> >>>> -- >>>> ----- >>>> Pozdrawiam >>>> >>>> Sławek "sZiBis" Skowron >>> >>> >>> >>> -- >>> ----- >>> Pozdrawiam >>> >>> Sławek "sZiBis" Skowron -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Solved] Re: Access Dienied for bucket upload - 403 code 2012-09-11 23:27 ` Yehuda Sadeh 2012-09-12 5:07 ` Sławomir Skowron @ 2012-09-12 14:21 ` Sławomir Skowron 1 sibling, 0 replies; 16+ messages in thread From: Sławomir Skowron @ 2012-09-12 14:21 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: ceph-devel@vger.kernel.org Problem solved. Everything because Nginx, and a request_uri in fcgi params to radosgw. Now request_uri is ok, and problem disappear. Big thanks for help Yehuda. Regards Dnia 12 wrz 2012 o godz. 01:27 Yehuda Sadeh <yehuda@inktank.com> napisał(a): > On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron <szibis@gmail.com> wrote: >> And more logs: >> >> >> 2012-09-11 21:03:38.357304 7faf0bf4f700 1 ====== req done >> req=0x141a650 http_status=403 ====== >> 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 >> 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty >> 2012-09-11 21:23:54.423198 7faf0bf4f700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.000039::::initializing >> 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s->object=<NULL> s->bucket=<NULL> >> 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ >> 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ >> 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ >> 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 >> 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 >> 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 >> 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 >> ...skipping... >> 2012-09-11 22:23:44.530567 7faf0bf4f700 10 >> s->object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> s->bucket=ocdn >> 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= >> 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET >> 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= >> 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= >> 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= >> 2012-09-11 22:23:44.530595 7faf0bf4f700 20 >> SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530596 7faf0bf4f700 20 >> REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530598 7faf0bf4f700 20 >> DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 >> 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 >> 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 >> 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= >> 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 >> 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive >> 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 >> (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like >> Gecko) Chrome/21.0.1180.89 Safari/537.1 >> 2012-09-11 22:23:44.530615 7faf0bf4f700 20 >> HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 >> 2012-09-11 22:23:44.530616 7faf0bf4f700 20 >> HTTP_ACCEPT_ENCODING=gzip,deflate,sdch >> 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 >> 2012-09-11 22:23:44.530618 7faf0bf4f700 20 >> HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 >> 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting >> op >> 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing >> 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading >> permissions >> 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531746 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.531760 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531767 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=.rgw:ocdn state=0x7fb01c013248 >> s->prefetch_data=0 >> 2012-09-11 22:23:44.531771 7faf0bf4f700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:23:44.531773 7faf0bf4f700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:23:44.532701 7faf0bf4f700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:23:44.532721 7faf0bf4f700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:23:44.532734 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 obj=ocdn: state=0x7fb01c013248 s->prefetch_data=0 >> 2012-09-11 22:23:44.532750 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.532770 7faf0bf4f700 20 get_obj_state: >> rctx=0x7fb01c0122c0 >> obj=ocdn:images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg >> state=0x7fb01c00a608 s->prefetch_data=1 >> 2012-09-11 22:23:44.534298 7faf0bf4f700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:23:44.534315 7faf0bf4f700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:23:44.534327 7faf0bf4f700 2 req 68518:0.003823:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:verifying >> op permissions >> 2012-09-11 22:23:44.534334 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=1 >> 2012-09-11 22:23:44.534336 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534337 7faf0bf4f700 5 Searching permissions for >> group=1 mask=1 >> 2012-09-11 22:23:44.534338 7faf0bf4f700 5 Permissions for group not found >> 2012-09-11 22:23:44.534339 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534341 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=1, policy perm=0, user_perm_mask=15, acl perm=0 >> 2012-09-11 22:23:44.534343 7faf0bf4f700 5 Searching permissions for >> uid=anonymous mask=16 >> 2012-09-11 22:23:44.534344 7faf0bf4f700 5 Permissions for user not found >> 2012-09-11 22:23:44.534345 7faf0bf4f700 5 Searching permissions for >> group=1 mask=16 >> 2012-09-11 22:23:44.534347 7faf0bf4f700 5 Found permission: 1 >> 2012-09-11 22:23:44.534348 7faf0bf4f700 5 Getting permissions >> id=anonymous owner=0 perm=0 >> 2012-09-11 22:23:44.534349 7faf0bf4f700 10 uid=anonymous requested >> perm (type)=16, policy perm=0, user_perm_mask=16, acl perm=0 >> 2012-09-11 22:23:44.534357 7faf0bf4f700 10 --> Status: 403 >> 2012-09-11 22:23:44.534367 7faf0bf4f700 10 --> Content-Length: 78 >> 2012-09-11 22:23:44.534370 7faf0bf4f700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:23:44.534373 7faf0bf4f700 10 --> Content-type: application/xml >> 2012-09-11 22:23:44.534480 7faf0bf4f700 2 req 68518:0.003976:s3:GET >> /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:http >> status=403 >> 2012-09-11 22:23:44.534765 7faf0bf4f700 1 ====== req done >> req=0x13995c0 http_status=403 ====== >> >> Good put, and 403 after: >> >> 2012-09-11 22:37:34.677422 7fafddffb700 1 ====== starting new request >> req=0x139a3d0 ===== >> 2012-09-11 22:37:34.677466 7fafddffb700 2 req 71187:0.000044::::initializing >> 2012-09-11 22:37:34.677492 7fafddffb700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.677501 7fafddffb700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.677502 7fafddffb700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.677503 7fafddffb700 20 QUERY_STRING= >> 2012-09-11 22:37:34.677504 7fafddffb700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.677505 7fafddffb700 20 CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677506 7fafddffb700 20 CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677507 7fafddffb700 20 HTTP_CONTENT_LENGTH=4721 >> 2012-09-11 22:37:34.677508 7fafddffb700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677509 7fafddffb700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677510 7fafddffb700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.677511 7fafddffb700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.677512 7fafddffb700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.677512 7fafddffb700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.677513 7fafddffb700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.677514 7fafddffb700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.677515 7fafddffb700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.677516 7fafddffb700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.677517 7fafddffb700 20 SERVER_NAME= >> 2012-09-11 22:37:34.677518 7fafddffb700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.677519 7fafddffb700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.677520 7fafddffb700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.677521 7fafddffb700 20 >> HTTP_CONTENT_MD5=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.677522 7fafddffb700 20 HTTP_EXPECT=100-Continue >> 2012-09-11 22:37:34.677523 7fafddffb700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.677524 7fafddffb700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.677525 7fafddffb700 20 HTTP_CONTENT_TYPE=text/plain >> 2012-09-11 22:37:34.677526 7fafddffb700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.677529 7fafddffb700 2 req 71187:0.000108:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.677534 7fafddffb700 2 req 71187:0.000113:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.677552 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.679069 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.679086 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680427 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.680441 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c004960 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb00c004a18 s->prefetch_data=0 >> 2012-09-11 22:37:34.680452 7fafddffb700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.680457 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.681685 7fafddffb700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.681748 7fafddffb700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.681766 7fafddffb700 10 auth_hdr: > > missing some lines here, did you grep the output? > >> 2012-09-11 22:37:34.681911 7fafddffb700 15 b64=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681921 7fafddffb700 15 >> auth_sign=5ucLfRdtJtAdcIcns2pgMowWAEE= >> 2012-09-11 22:37:34.681922 7fafddffb700 15 compare=0 >> 2012-09-11 22:37:34.681926 7fafddffb700 2 req 71187:0.004504:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:reading >> permissions >> 2012-09-11 22:37:34.681941 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683575 7fafddffb700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.683588 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683596 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=.rgw:ocdn state=0x7fb00c006578 >> s->prefetch_data=0 >> 2012-09-11 22:37:34.683599 7fafddffb700 20 state for obj=.rgw:ocdn is >> not atomic, not appending atomic test >> 2012-09-11 22:37:34.683601 7fafddffb700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.684735 7fafddffb700 20 rados->read r=0 bl.length=65 >> 2012-09-11 22:37:34.684755 7fafddffb700 20 rgw_get_bucket_info: >> bucket=ocdn(@.rgw.buckets[4168.2]) owner 0 >> 2012-09-11 22:37:34.684769 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 obj=ocdn: state=0x7fb00c006578 s->prefetch_data=0 >> 2012-09-11 22:37:34.684786 7fafddffb700 15 Read >> AccessControlPolicy<AccessControlPolicy >> xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0</ID><DisplayName>ocdn</DisplayName></Owner><AccessControlList><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:type="CanonicalUser"><ID>0</ID><DisplayName>ocdn</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> >> 2012-09-11 22:37:34.684796 7fafddffb700 2 req 71187:0.007375:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op permissions >> 2012-09-11 22:37:34.684801 7fafddffb700 5 Searching permissions for >> uid=0 mask=2 >> 2012-09-11 22:37:34.684803 7fafddffb700 5 Found permission: 15 >> 2012-09-11 22:37:34.684804 7fafddffb700 10 uid=0 requested perm >> (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 >> 2012-09-11 22:37:34.684808 7fafddffb700 2 req 71187:0.007387:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:verifying >> op params >> 2012-09-11 22:37:34.684811 7fafddffb700 2 req 71187:0.007390:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:executing >> 2012-09-11 22:37:34.684831 7fafddffb700 15 >> supplied_md5_b64=7Ous3d6VIkuW9GMzkSBJsQ== >> 2012-09-11 22:37:34.684834 7fafddffb700 15 ceph_armor ret=16 >> 2012-09-11 22:37:34.684841 7fafddffb700 15 >> supplied_md5=ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.684901 7fafddffb700 20 get_obj_state: >> rctx=0x7fb00c007050 >> obj=ocdn:files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> state=0x7fb00c00bda8 s->prefetch_data=0 >> 2012-09-11 22:37:34.686278 7fafddffb700 20 >> prepare_atomic_for_write_impl: state is not atomic. >> state=0x7fb00c00bda8 >> 2012-09-11 22:37:34.704899 7fafddffb700 10 --> ETag: "ecebacddde95224b96f46333 >> 2012-09-11 22:37:34.704949 7fafddffb700 10 --> Content-Length: 0 >> 2012-09-11 22:37:34.704954 7fafddffb700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:37:34.704958 7fafddffb700 10 --> Status: 200 >> 2012-09-11 22:37:34.704963 7fafddffb700 10 --> Content-type: application/xml >> 2012-09-11 22:37:34.705099 7fafddffb700 2 req 71187:0.027678:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >> status=200 >> 2012-09-11 22:37:34.705454 7fafddffb700 1 ====== req done >> req=0x139a3d0 http_status=200 ====== >> >> >> 2012-09-11 22:37:34.708316 7faec76c6700 20 dequeued request req=0x1368860 >> 2012-09-11 22:37:34.708326 7faec76c6700 20 RGWWQ: empty >> 2012-09-11 22:37:34.708333 7faec76c6700 1 ====== starting new request >> req=0x1368860 ===== >> 2012-09-11 22:37:34.708375 7faec76c6700 2 req 71188:0.000042::::initializing >> 2012-09-11 22:37:34.708402 7faec76c6700 10 >> s->object=files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> s->bucket=ocdn >> 2012-09-11 22:37:34.708408 7faec76c6700 10 meta>> HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708425 7faec76c6700 10 x>> x-amz-acl:public-read >> 2012-09-11 22:37:34.708431 7faec76c6700 20 FCGI_ROLE=RESPONDER >> 2012-09-11 22:37:34.708432 7faec76c6700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >> 2012-09-11 22:37:34.708434 7faec76c6700 20 QUERY_STRING=acl >> 2012-09-11 22:37:34.708435 7faec76c6700 20 REQUEST_METHOD=PUT >> 2012-09-11 22:37:34.708437 7faec76c6700 20 CONTENT_TYPE= >> 2012-09-11 22:37:34.708438 7faec76c6700 20 CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708439 7faec76c6700 20 HTTP_CONTENT_LENGTH=0 >> 2012-09-11 22:37:34.708440 7faec76c6700 20 >> SCRIPT_NAME=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> REQUEST_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708441 7faec76c6700 20 >> DOCUMENT_URI=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.708442 7faec76c6700 20 DOCUMENT_ROOT=/var/www >> 2012-09-11 22:37:34.708443 7faec76c6700 20 SERVER_PROTOCOL=HTTP/1.1 >> 2012-09-11 22:37:34.708444 7faec76c6700 20 GATEWAY_INTERFACE=CGI/1.1 >> 2012-09-11 22:37:34.708446 7faec76c6700 20 SERVER_SOFTWARE=nginx/1.2.0 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_ADDR=10.174.53.77 >> 2012-09-11 22:37:34.708447 7faec76c6700 20 REMOTE_PORT=48588 >> 2012-09-11 22:37:34.708448 7faec76c6700 20 SERVER_ADDR=10.177.0.3 >> 2012-09-11 22:37:34.708450 7faec76c6700 20 SERVER_PORT=80 >> 2012-09-11 22:37:34.708451 7faec76c6700 20 SERVER_NAME= >> 2012-09-11 22:37:34.708452 7faec76c6700 20 REDIRECT_STATUS=200 >> 2012-09-11 22:37:34.708453 7faec76c6700 20 RGW_SHOULD_LOG=no >> 2012-09-11 22:37:34.708454 7faec76c6700 20 HTTP_HOST=10.177.0.3 >> 2012-09-11 22:37:34.708455 7faec76c6700 20 HTTP_ACCEPT_ENCODING=identity >> 2012-09-11 22:37:34.708456 7faec76c6700 20 HTTP_DATE=Tue, 11 Sep 2012 >> 20:37:46 GMT >> 2012-09-11 22:37:34.708457 7faec76c6700 20 HTTP_X_AMZ_ACL=public-read >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_AUTHORIZATION=AWS >> 73VF66Q2JH5PT5K7QJ3A:xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.708458 7faec76c6700 20 HTTP_USER_AGENT=Boto/2.5.1 (linux2) >> 2012-09-11 22:37:34.708462 7faec76c6700 2 req 71188:0.000129:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1::getting >> op >> 2012-09-11 22:37:34.708467 7faec76c6700 2 req 71188:0.000134:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:authorizing >> 2012-09-11 22:37:34.708486 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.709812 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.709828 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711091 7faec76c6700 20 get_obj_state: s->obj_tag >> was set empty >> 2012-09-11 22:37:34.711101 7faec76c6700 20 get_obj_state: >> rctx=0x7fb2a4007e60 obj=.users:73VF66Q2JH5PT5K7QJ3A >> state=0x7fb2a4007f18 s->prefetch_data=0 >> 2012-09-11 22:37:34.711108 7faec76c6700 20 state for >> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >> test >> 2012-09-11 22:37:34.711111 7faec76c6700 20 rados->read obj-ofs=0 >> read_ofs=0 read_len=16384 >> 2012-09-11 22:37:34.712485 7faec76c6700 20 rados->read r=0 bl.length=231 >> 2012-09-11 22:37:34.712655 7faec76c6700 10 get_canon_resource(): >> dest=/ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1 >> 2012-09-11 22:37:34.712659 7faec76c6700 10 auth_hdr: > > and here also > >> 2012-09-11 22:37:34.712735 7faec76c6700 15 b64=7pIUzzJaX/EyTV89LMXwcqvS06w= >> 2012-09-11 22:37:34.712742 7faec76c6700 15 >> auth_sign=xHPA2K5QvuehAvKH9WdOvf+ZKzE= >> 2012-09-11 22:37:34.712744 7faec76c6700 15 compare=65 >> 2012-09-11 22:37:34.712747 7faec76c6700 10 failed to authorize request >> 2012-09-11 22:37:34.712754 7faec76c6700 10 --> Status: 403 >> 2012-09-11 22:37:34.712766 7faec76c6700 10 --> Content-Length: 78 >> 2012-09-11 22:37:34.712769 7faec76c6700 10 --> Accept-Ranges: bytes >> 2012-09-11 22:37:34.712772 7faec76c6700 10 --> Content-type: application/xml >> 2012-09-11 22:37:34.712887 7faec76c6700 2 req 71188:0.004554:s3:PUT >> /ocdn/files/pulscms/MjU7MDA_/ecebacddde95224b96f46333912049b1:put_obj:http >> status=403 >> 2012-09-11 22:37:34.713093 7faec76c6700 1 ====== req done >> req=0x1368860 http_status=403 ====== >> >> On Tue, Sep 11, 2012 at 7:46 PM, Sławomir Skowron <szibis@gmail.com> wrote: >>> On Tue, Sep 11, 2012 at 6:48 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>> On Tue, Sep 11, 2012 at 9:45 AM, Yehuda Sadeh <yehuda@inktank.com> wrote: >>>>> On Tue, Sep 11, 2012 at 7:28 AM, Sławomir Skowron <szibis@gmail.com> wrote: >>>>>> Every acl operation ending with 403 in PUT. >>>>>> >>>>>> ~# s3 -u test oc >>>>>> Bucket Status >>>>>> -------------------------------------------------------- -------------------- >>>>>> oc Access Denied >>>>>> >>>>>> Anyone know why, and how to enable this bucket ?? Now i have problems >>>>>> with cluster, because there is no way to upload new file >>>>>> >>>>>> ~# s3 -u getacl oc >>>>>> >>>>>> ERROR: ErrorAccessDenied >>>>>> >>>>> >>>>> User somehow lost bucket ownership (was it actually the owner?). Do >>>>> you know how to reproduce the issue? any remaining logs? >>>>> >>>>> Try getting bucket info: >>>>> >>>>> # radosgw-admin bucket stats --bucket=oc >>>>> >>>>> If that doesn't fail and actually shows relevant info, try checking >>>>> whether the user credentials match the s3 tool credentials. >>>>> >>>> Oh, and thinking about it some more.. 'oc' is a too short name for a >>>> bucket (requires min of 3 chars). How did you create it? The failure >>>> may be related. >>> >>> Yes i made a shortcut of name :)) >>> >>> Right now every bucket in pool, are afected >>> >>> :~#radosgw-admin bucket stats --bucket=lvstest >>> { "bucket": "lvstest", >>> "pool": ".rgw.buckets", >>> "id": "1142048.1", >>> "marker": "1142048.1", >>> "owner": "0", >>> "usage": { "rgw.main": { "size_kb": 1, >>> "size_kb_actual": 4, >>> "num_objects": 1}}} >>> :~# radosgw-admin bucket stats --bucket=ocdn >>> { "bucket": "ocdn", >>> "pool": ".rgw.buckets", >>> "id": "4168.2", >>> "marker": "4168.2", >>> "owner": "0", >>> "usage": { "rgw.main": { "size_kb": 513059717, >>> "size_kb_actual": 516402364, >>> "num_objects": 1606730}}} >>> >>> Credentials from radosgw-admin user info match that from clients requests. >>> >>> Every GET, PUT, HEAD using this credentials works fine, but only one >>> operations does not work (403 from radosgw) - setting acl for object >>> for a public-read. Setting canned acl with PUT for public-read from >>> s3lib work good, but get/set acl failed. >>> >>> list bucket object works good, and list buckets via s3 client. >>> >>> Now i can't reproduce, but i will dig logs from radosgw, for related >>> time, when this happend. >>> >>> Example 403 from radosgw log, before that PUT of object ends with 200: >>> >>> 2012-09-11 19:36:34.346312 7fb25d7fa700 1 ====== req done >>> req=0x1435980 http_status=403 ====== >>> 2012-09-11 19:37:04.342894 7fb25d7fa700 20 dequeued request req=0x13994c0 >>> 2012-09-11 19:37:04.342903 7fb25d7fa700 20 RGWWQ: empty >>> 2012-09-11 19:37:04.342910 7fb25d7fa700 1 ====== starting new request >>> req=0x13994c0 ===== >>> 2012-09-11 19:37:04.342948 7fb25d7fa700 2 req 39665:0.000038::::initializing >>> 2012-09-11 19:37:04.342971 7fb25d7fa700 10 >>> s->object=images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> s->bucket=ocdn >>> 2012-09-11 19:37:04.342983 7fb25d7fa700 10 meta>> HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 19:37:04.342991 7fb25d7fa700 10 x>> x-amz-acl:public-read >>> 2012-09-11 19:37:04.342996 7fb25d7fa700 20 FCGI_ROLE=RESPONDER >>> 2012-09-11 19:37:04.342997 7fb25d7fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi >>> 2012-09-11 19:37:04.342999 7fb25d7fa700 20 QUERY_STRING=acl >>> 2012-09-11 19:37:04.343001 7fb25d7fa700 20 REQUEST_METHOD=PUT >>> 2012-09-11 19:37:04.343002 7fb25d7fa700 20 CONTENT_TYPE= >>> 2012-09-11 19:37:04.343003 7fb25d7fa700 20 CONTENT_LENGTH=0 >>> 2012-09-11 19:37:04.343004 7fb25d7fa700 20 HTTP_CONTENT_LENGTH=0 >>> 2012-09-11 19:37:04.343005 7fb25d7fa700 20 >>> SCRIPT_NAME=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343006 7fb25d7fa700 20 >>> REQUEST_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343007 7fb25d7fa700 20 >>> DOCUMENT_URI=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.343008 7fb25d7fa700 20 DOCUMENT_ROOT=/var/www >>> 2012-09-11 19:37:04.343009 7fb25d7fa700 20 SERVER_PROTOCOL=HTTP/1.1 >>> 2012-09-11 19:37:04.343010 7fb25d7fa700 20 GATEWAY_INTERFACE=CGI/1.1 >>> 2012-09-11 19:37:04.343011 7fb25d7fa700 20 SERVER_SOFTWARE=nginx/1.2.0 >>> 2012-09-11 19:37:04.343012 7fb25d7fa700 20 REMOTE_ADDR=10.177.62.9 >>> 2012-09-11 19:37:04.343013 7fb25d7fa700 20 REMOTE_PORT=56378 >>> 2012-09-11 19:37:04.343014 7fb25d7fa700 20 SERVER_ADDR=10.177.0.3 >>> 2012-09-11 19:37:04.343015 7fb25d7fa700 20 SERVER_PORT=80 >>> 2012-09-11 19:37:04.343016 7fb25d7fa700 20 SERVER_NAME= >>> 2012-09-11 19:37:04.343017 7fb25d7fa700 20 REDIRECT_STATUS=200 >>> 2012-09-11 19:37:04.343018 7fb25d7fa700 20 RGW_SHOULD_LOG=no >>> 2012-09-11 19:37:04.343019 7fb25d7fa700 20 HTTP_HOST=10.177.0.3 >>> 2012-09-11 19:37:04.343020 7fb25d7fa700 20 HTTP_ACCEPT_ENCODING=identity >>> 2012-09-11 19:37:04.343021 7fb25d7fa700 20 HTTP_DATE=Tue, 11 Sep 2012 >>> 17:37:02 GMT >>> 2012-09-11 19:37:04.343022 7fb25d7fa700 20 HTTP_X_AMZ_ACL=public-read >>> 2012-09-11 19:37:04.343023 7fb25d7fa700 20 HTTP_AUTHORIZATION=AWS >>> 73VF66Q2JH5PT5K7QJ3A:mh7yBIFGPtdI1MAjW39/1ywV580= >>> 2012-09-11 19:37:04.343024 7fb25d7fa700 20 HTTP_USER_AGENT=Boto/2.5.2 (linux2) >>> 2012-09-11 19:37:04.343026 7fb25d7fa700 2 req 39665:0.000116:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg::getting >>> op >>> 2012-09-11 19:37:04.343031 7fb25d7fa700 2 req 39665:0.000121:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:authorizing >>> 2012-09-11 19:37:04.343047 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.344527 7fb25d7fa700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 19:37:04.344542 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.345847 7fb25d7fa700 20 get_obj_state: s->obj_tag >>> was set empty >>> 2012-09-11 19:37:04.345858 7fb25d7fa700 20 get_obj_state: >>> rctx=0x7fb23c006340 obj=.users:73VF66Q2JH5PT5K7QJ3A >>> state=0x7fb23c00b468 s->prefetch_data=0 >>> 2012-09-11 19:37:04.345863 7fb25d7fa700 20 state for >>> obj=.users:73VF66Q2JH5PT5K7QJ3A is not atomic, not appending atomic >>> test >>> 2012-09-11 19:37:04.345866 7fb25d7fa700 20 rados->read obj-ofs=0 >>> read_ofs=0 read_len=16384 >>> 2012-09-11 19:37:04.347069 7fb25d7fa700 20 rados->read r=0 bl.length=231 >>> 2012-09-11 19:37:04.347121 7fb25d7fa700 10 get_canon_resource(): >>> dest=/ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg >>> 2012-09-11 19:37:04.347126 7fb25d7fa700 10 auth_hdr: >>> 2012-09-11 19:37:04.347195 7fb25d7fa700 15 b64=9qJQCGmPbTXcT6a4qD3kED0PPdY= >>> 2012-09-11 19:37:04.347198 7fb25d7fa700 15 >>> auth_sign=mh7yBIFGPtdI1MAjW39/1ywV580= >>> 2012-09-11 19:37:04.347199 7fb25d7fa700 15 compare=52 >>> 2012-09-11 19:37:04.347201 7fb25d7fa700 10 failed to authorize request >>> 2012-09-11 19:37:04.347208 7fb25d7fa700 10 --> Status: 403 >>> 2012-09-11 19:37:04.347218 7fb25d7fa700 10 --> Content-Length: 78 >>> 2012-09-11 19:37:04.347220 7fb25d7fa700 10 --> Accept-Ranges: bytes >>> 2012-09-11 19:37:04.347223 7fb25d7fa700 10 --> Content-type: application/xml >>> 2012-09-11 19:37:04.347346 7fb25d7fa700 2 req 39665:0.004436:s3:PUT >>> /ocdn/images/pulscms/ZjM7MDA_/d6d6df3de5afa365d0fb7379fdbd75b8.jpg:put_obj:http >>> status=403 >>> 2012-09-11 19:37:04.347600 7fb25d7fa700 1 ====== req done >>> req=0x13994c0 http_status=403 ====== >>> 2012-09-11 19:40:04.354709 7fb25d7fa700 20 dequeued request req=0x13994c0 >>> >>> Interesting is that second cluster, which is replicated via s3 client, >>> get the same issue, from that in production one. Is this can be >>> generated by many error on radosgw layer ?? >>> >>> I will try to find beginning of this problems in logs. >>> >>>> >>>> Yehuda >>> >>> >>> >>> -- >>> ----- >>> Pozdrawiam >>> >>> Sławek "sZiBis" Skowron >> >> >> >> -- >> ----- >> Pozdrawiam >> >> Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2012-09-12 14:22 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-09-11 14:28 Access Dienied for bucket upload - 403 code Sławomir Skowron 2012-09-11 16:45 ` Yehuda Sadeh 2012-09-11 16:48 ` Yehuda Sadeh 2012-09-11 17:46 ` Sławomir Skowron 2012-09-11 20:41 ` Sławomir Skowron 2012-09-11 20:46 ` Yehuda Sadeh 2012-09-11 21:28 ` Sławomir Skowron 2012-09-11 21:32 ` Yehuda Sadeh 2012-09-11 21:38 ` Sławomir Skowron 2012-09-11 21:44 ` Yehuda Sadeh 2012-09-11 21:50 ` Sławomir Skowron 2012-09-11 21:55 ` Yehuda Sadeh 2012-09-11 23:27 ` Yehuda Sadeh 2012-09-12 5:07 ` Sławomir Skowron 2012-09-12 9:14 ` Sławomir Skowron 2012-09-12 14:21 ` [Solved] " Sławomir Skowron
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.