radosgw + s3 + keystone + Browser-Based POST bug

radosgw + s3 + keystone + Browser-Based POST bug

[Valery Tschopp]

[-- Attachment #1.1: Type: text/plain, Size: 1665 bytes --]

Hi guys,

We have integrated our radosgw (v0.80.7) with our OpenStack Keystone 
server (icehouse) successfully.

The "normal" S3 operations can be executed with the Keystone user's EC2 
credentials (EC2_ACCESS_KEY, EC2_SECRET_KEY). The radosgw correctly 
handles these user credentials, ask keystone to validate them, and the 
resulting objects belong to the Keystone tenant/project or the user 
(user is member of the tenant/project).

But for the "Browser-based upload POST" [1] it doesn't work! The user is 
not correctly resolved, and the radosgw returns a 403 code!

It looks like the s3 keystone integration doesn't work correctly when a 
S3 browser-based upload POST is used.

See the attached log file (radosgw.log), you can clearly see the user 
lookup failing, and the status being set to 403:


2015-01-29 15:11:30.151157 7f25616fa700  0 User lookup failed!
2015-01-29 15:11:30.151171 7f25616fa700 15 Read 
RGWCORSConfiguration<CORSConfiguration><CORSRule><AllowedMethod>POST</AllowedMethod><AllowedOrigin>https://staging.tube.switch.ch</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
2015-01-29 15:11:30.151184 7f25616fa700 10 Method POST is supported
2015-01-29 15:11:30.151195 7f25616fa700  2 req 1123:0.013204:s3:POST 
/:post_obj:http status=403


Is this a bug? Or did we miss something else?

Cheers,
Valery

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html
-- 
SWITCH
--------------------------
Valery Tschopp, Software Engineer, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
email: valery.tschopp@switch.ch phone: +41 44 268 1544



[-- Attachment #1.2: radosgw.log --]
[-- Type: text/plain, Size: 18389 bytes --]

2015-01-29 15:11:30.130054 7f2634cef700 20 enqueued request req=0x7f26040838d0
2015-01-29 15:11:30.130084 7f2634cef700 20 RGWWQ:
2015-01-29 15:11:30.130086 7f2634cef700 20 req: 0x7f26040838d0
2015-01-29 15:11:30.130108 7f2634cef700 10 allocated request req=0x7f26040c58d0
2015-01-29 15:11:30.130200 7f2454ce1700 20 dequeued request req=0x7f26040838d0
2015-01-29 15:11:30.130208 7f2454ce1700 20 RGWWQ: empty
2015-01-29 15:11:30.130303 7f2454ce1700 20 CONTEXT_DOCUMENT_ROOT=/var/www
2015-01-29 15:11:30.130305 7f2454ce1700 20 CONTEXT_PREFIX=
2015-01-29 15:11:30.130306 7f2454ce1700 20 DOCUMENT_ROOT=/var/www
2015-01-29 15:11:30.130307 7f2454ce1700 20 FCGI_ROLE=RESPONDER
2015-01-29 15:11:30.130308 7f2454ce1700 20 GATEWAY_INTERFACE=CGI/1.1
2015-01-29 15:11:30.130308 7f2454ce1700 20 HTTP_ACCEPT=*/*
2015-01-29 15:11:30.130309 7f2454ce1700 20 HTTP_ACCEPT_ENCODING=gzip, deflate, sdch
2015-01-29 15:11:30.130310 7f2454ce1700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8,it;q=0.6
2015-01-29 15:11:30.130311 7f2454ce1700 20 HTTP_ACCESS_CONTROL_REQUEST_HEADERS=content-type
2015-01-29 15:11:30.130312 7f2454ce1700 20 HTTP_ACCESS_CONTROL_REQUEST_METHOD=POST
2015-01-29 15:11:30.130312 7f2454ce1700 20 HTTP_AUTHORIZATION=
2015-01-29 15:11:30.130313 7f2454ce1700 20 HTTP_CACHE_CONTROL=no-cache
2015-01-29 15:11:30.130314 7f2454ce1700 20 HTTP_CONNECTION=keep-alive
2015-01-29 15:11:30.130314 7f2454ce1700 20 HTTP_HOST=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.130315 7f2454ce1700 20 HTTP_ORIGIN=https://staging.tube.switch.ch
2015-01-29 15:11:30.130316 7f2454ce1700 20 HTTP_PRAGMA=no-cache
2015-01-29 15:11:30.130317 7f2454ce1700 20 HTTP_REFERER=https://staging.tube.switch.ch/channels/04238519/videos
2015-01-29 15:11:30.130318 7f2454ce1700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
2015-01-29 15:11:30.130320 7f2454ce1700 20 HTTPS=on
2015-01-29 15:11:30.130321 7f2454ce1700 20 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2015-01-29 15:11:30.130322 7f2454ce1700 20 QUERY_STRING=
2015-01-29 15:11:30.130322 7f2454ce1700 20 REMOTE_ADDR=130.59.17.201
2015-01-29 15:11:30.130323 7f2454ce1700 20 REMOTE_PORT=53901
2015-01-29 15:11:30.130324 7f2454ce1700 20 REQUEST_METHOD=OPTIONS
2015-01-29 15:11:30.130325 7f2454ce1700 20 REQUEST_SCHEME=https
2015-01-29 15:11:30.130326 7f2454ce1700 20 REQUEST_URI=/
2015-01-29 15:11:30.130327 7f2454ce1700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi
2015-01-29 15:11:30.130328 7f2454ce1700 20 SCRIPT_NAME=/
2015-01-29 15:11:30.130329 7f2454ce1700 20 SCRIPT_URI=https://switch-original-staging.os.zhdk.cloud.switch.ch/
2015-01-29 15:11:30.130330 7f2454ce1700 20 SCRIPT_URL=/
2015-01-29 15:11:30.130331 7f2454ce1700 20 SERVER_ADDR=86.119.32.13
2015-01-29 15:11:30.130332 7f2454ce1700 20 SERVER_ADMIN=cloud@switch.ch
2015-01-29 15:11:30.130333 7f2454ce1700 20 SERVER_NAME=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.130334 7f2454ce1700 20 SERVER_PORT=443
2015-01-29 15:11:30.130334 7f2454ce1700 20 SERVER_PROTOCOL=HTTP/1.1
2015-01-29 15:11:30.130335 7f2454ce1700 20 SERVER_SIGNATURE=
2015-01-29 15:11:30.130350 7f2454ce1700 20 SERVER_SOFTWARE=Apache/2.4.7 (Ubuntu)
2015-01-29 15:11:30.130351 7f2454ce1700 20 SSL_TLS_SNI=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.130352 7f2454ce1700  1 ====== starting new request req=0x7f26040838d0 =====
2015-01-29 15:11:30.130403 7f2454ce1700  2 req 1122:0.000050::OPTIONS /::initializing
2015-01-29 15:11:30.130414 7f2454ce1700 10 host=switch-original-staging.os.zhdk.cloud.switch.ch rgw_dns_name=os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.130486 7f2454ce1700 10 s->object=<NULL> s->bucket=switch-original-staging
2015-01-29 15:11:30.130499 7f2454ce1700  2 req 1122:0.000147:s3:OPTIONS /::getting op
2015-01-29 15:11:30.130504 7f2454ce1700  2 req 1122:0.000152:s3:OPTIONS /:options_cors:authorizing
2015-01-29 15:11:30.130522 7f2454ce1700  2 req 1122:0.000170:s3:OPTIONS /:options_cors:reading permissions
2015-01-29 15:11:30.130672 7f2454ce1700 20 get_obj_state: rctx=0x7f2454ce0250 obj=.rgw:switch-original-staging state=0x7f26300142c8 s->prefetch_data=0
2015-01-29 15:11:30.130703 7f2454ce1700 10 cache get: name=.rgw+switch-original-staging : hit
2015-01-29 15:11:30.130714 7f2454ce1700 20 get_obj_state: s->obj_tag was set empty
2015-01-29 15:11:30.130722 7f2454ce1700 20 Read xattr: user.rgw.idtag
2015-01-29 15:11:30.130724 7f2454ce1700 20 Read xattr: user.rgw.manifest
2015-01-29 15:11:30.130729 7f2454ce1700 10 cache get: name=.rgw+switch-original-staging : hit
2015-01-29 15:11:30.130755 7f2454ce1700 20 rgw_get_bucket_info: bucket instance: switch-original-staging(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.4063334.16])
2015-01-29 15:11:30.130767 7f2454ce1700 20 reading from .rgw:.bucket.meta.switch-original-staging:default.4063334.16
2015-01-29 15:11:30.130782 7f2454ce1700 20 get_obj_state: rctx=0x7f2454ce0250 obj=.rgw:.bucket.meta.switch-original-staging:default.4063334.16 state=0x7f26300251d8 s->prefetch_data=0
2015-01-29 15:11:30.130791 7f2454ce1700 10 cache get: name=.rgw+.bucket.meta.switch-original-staging:default.4063334.16 : hit
2015-01-29 15:11:30.130809 7f2454ce1700 20 get_obj_state: s->obj_tag was set empty
2015-01-29 15:11:30.130812 7f2454ce1700 20 Read xattr: user.rgw.acl
2015-01-29 15:11:30.130813 7f2454ce1700 20 Read xattr: user.rgw.cors
2015-01-29 15:11:30.130814 7f2454ce1700 20 Read xattr: user.rgw.idtag
2015-01-29 15:11:30.130815 7f2454ce1700 20 Read xattr: user.rgw.manifest
2015-01-29 15:11:30.130817 7f2454ce1700 10 cache get: name=.rgw+.bucket.meta.switch-original-staging:default.4063334.16 : hit
2015-01-29 15:11:30.130853 7f2454ce1700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>f2867c1c7dbd4b3b95ee616965ebabc0</ID><DisplayName>SWITCHtube-staging</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>f2867c1c7dbd4b3b95ee616965ebabc0</ID><DisplayName>SWITCHtube-staging</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-01-29 15:11:30.130868 7f2454ce1700  2 req 1122:0.000516:s3:OPTIONS /:options_cors:init op
2015-01-29 15:11:30.130875 7f2454ce1700  2 req 1122:0.000523:s3:OPTIONS /:options_cors:verifying op mask
2015-01-29 15:11:30.130881 7f2454ce1700 20 required_mask= 1 user.op_mask=7
2015-01-29 15:11:30.130882 7f2454ce1700  2 req 1122:0.000530:s3:OPTIONS /:options_cors:verifying op permissions
2015-01-29 15:11:30.130884 7f2454ce1700  2 req 1122:0.000532:s3:OPTIONS /:options_cors:verifying op params
2015-01-29 15:11:30.130886 7f2454ce1700  2 req 1122:0.000533:s3:OPTIONS /:options_cors:executing
2015-01-29 15:11:30.130896 7f2454ce1700 15 Read RGWCORSConfiguration<CORSConfiguration><CORSRule><AllowedMethod>POST</AllowedMethod><AllowedOrigin>https://staging.tube.switch.ch</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
2015-01-29 15:11:30.130934 7f2454ce1700 10 Method POST is supported
2015-01-29 15:11:30.130961 7f2454ce1700  2 req 1122:0.000609:s3:OPTIONS /:options_cors:http status=200
2015-01-29 15:11:30.130966 7f2454ce1700  1 ====== req done req=0x7f26040838d0 http_status=200 ======
2015-01-29 15:11:30.137819 7f2634cef700 20 enqueued request req=0x7f26040c58d0
2015-01-29 15:11:30.137833 7f2634cef700 20 RGWWQ:
2015-01-29 15:11:30.137834 7f2634cef700 20 req: 0x7f26040c58d0
2015-01-29 15:11:30.137841 7f2634cef700 10 allocated request req=0x7f26040a2dd0
2015-01-29 15:11:30.137891 7f25616fa700 20 dequeued request req=0x7f26040c58d0
2015-01-29 15:11:30.137899 7f25616fa700 20 RGWWQ: empty
2015-01-29 15:11:30.137957 7f25616fa700 20 CONTENT_LENGTH=665714
2015-01-29 15:11:30.137959 7f25616fa700 20 CONTENT_TYPE=multipart/form-data; boundary=----WebKitFormBoundary1l6shWgvNaxvykeY
2015-01-29 15:11:30.137960 7f25616fa700 20 CONTEXT_DOCUMENT_ROOT=/var/www
2015-01-29 15:11:30.137961 7f25616fa700 20 CONTEXT_PREFIX=
2015-01-29 15:11:30.137962 7f25616fa700 20 DOCUMENT_ROOT=/var/www
2015-01-29 15:11:30.137963 7f25616fa700 20 FCGI_ROLE=RESPONDER
2015-01-29 15:11:30.137963 7f25616fa700 20 GATEWAY_INTERFACE=CGI/1.1
2015-01-29 15:11:30.137965 7f25616fa700 20 HTTP_ACCEPT=*/*
2015-01-29 15:11:30.137965 7f25616fa700 20 HTTP_ACCEPT_ENCODING=gzip, deflate
2015-01-29 15:11:30.137966 7f25616fa700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8,it;q=0.6
2015-01-29 15:11:30.137967 7f25616fa700 20 HTTP_AUTHORIZATION=
2015-01-29 15:11:30.137968 7f25616fa700 20 HTTP_CACHE_CONTROL=no-cache
2015-01-29 15:11:30.137969 7f25616fa700 20 HTTP_CONNECTION=keep-alive
2015-01-29 15:11:30.137970 7f25616fa700 20 HTTP_HOST=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.137971 7f25616fa700 20 HTTP_ORIGIN=https://staging.tube.switch.ch
2015-01-29 15:11:30.137972 7f25616fa700 20 HTTP_PRAGMA=no-cache
2015-01-29 15:11:30.137973 7f25616fa700 20 HTTP_REFERER=https://staging.tube.switch.ch/channels/04238519/videos
2015-01-29 15:11:30.137974 7f25616fa700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
2015-01-29 15:11:30.137974 7f25616fa700 20 HTTPS=on
2015-01-29 15:11:30.137975 7f25616fa700 20 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2015-01-29 15:11:30.137976 7f25616fa700 20 QUERY_STRING=
2015-01-29 15:11:30.137977 7f25616fa700 20 REMOTE_ADDR=130.59.17.201
2015-01-29 15:11:30.137978 7f25616fa700 20 REMOTE_PORT=53901
2015-01-29 15:11:30.137979 7f25616fa700 20 REQUEST_METHOD=POST
2015-01-29 15:11:30.137979 7f25616fa700 20 REQUEST_SCHEME=https
2015-01-29 15:11:30.137980 7f25616fa700 20 REQUEST_URI=/
2015-01-29 15:11:30.137981 7f25616fa700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi
2015-01-29 15:11:30.137982 7f25616fa700 20 SCRIPT_NAME=/
2015-01-29 15:11:30.137983 7f25616fa700 20 SCRIPT_URI=https://switch-original-staging.os.zhdk.cloud.switch.ch/
2015-01-29 15:11:30.137984 7f25616fa700 20 SCRIPT_URL=/
2015-01-29 15:11:30.137985 7f25616fa700 20 SERVER_ADDR=86.119.32.13
2015-01-29 15:11:30.137986 7f25616fa700 20 SERVER_ADMIN=cloud@switch.ch
2015-01-29 15:11:30.137987 7f25616fa700 20 SERVER_NAME=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.137988 7f25616fa700 20 SERVER_PORT=443
2015-01-29 15:11:30.137989 7f25616fa700 20 SERVER_PROTOCOL=HTTP/1.1
2015-01-29 15:11:30.137989 7f25616fa700 20 SERVER_SIGNATURE=
2015-01-29 15:11:30.137990 7f25616fa700 20 SERVER_SOFTWARE=Apache/2.4.7 (Ubuntu)
2015-01-29 15:11:30.137991 7f25616fa700 20 SSL_TLS_SNI=switch-original-staging.os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.137992 7f25616fa700  1 ====== starting new request req=0x7f26040c58d0 =====
2015-01-29 15:11:30.138000 7f25616fa700  2 req 1123:0.000009::POST /::initializing
2015-01-29 15:11:30.138003 7f25616fa700 10 host=switch-original-staging.os.zhdk.cloud.switch.ch rgw_dns_name=os.zhdk.cloud.switch.ch
2015-01-29 15:11:30.138036 7f25616fa700 10 s->object=<NULL> s->bucket=switch-original-staging
2015-01-29 15:11:30.138040 7f25616fa700  2 req 1123:0.000049:s3:POST /::getting op
2015-01-29 15:11:30.138048 7f25616fa700  2 req 1123:0.000057:s3:POST /:post_obj:authorizing
2015-01-29 15:11:30.138052 7f25616fa700  2 req 1123:0.000061:s3:POST /:post_obj:reading permissions
2015-01-29 15:11:30.138100 7f25616fa700 20 get_obj_state: rctx=0x7f25616f9250 obj=.rgw:switch-original-staging state=0x7f26383ffce8 s->prefetch_data=0
2015-01-29 15:11:30.138106 7f25616fa700 10 cache get: name=.rgw+switch-original-staging : hit
2015-01-29 15:11:30.138111 7f25616fa700 20 get_obj_state: s->obj_tag was set empty
2015-01-29 15:11:30.138112 7f25616fa700 20 Read xattr: user.rgw.idtag
2015-01-29 15:11:30.138113 7f25616fa700 20 Read xattr: user.rgw.manifest
2015-01-29 15:11:30.138116 7f25616fa700 10 cache get: name=.rgw+switch-original-staging : hit
2015-01-29 15:11:30.138125 7f25616fa700 20 rgw_get_bucket_info: bucket instance: switch-original-staging(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.4063334.16])
2015-01-29 15:11:30.138129 7f25616fa700 20 reading from .rgw:.bucket.meta.switch-original-staging:default.4063334.16
2015-01-29 15:11:30.138142 7f25616fa700 20 get_obj_state: rctx=0x7f25616f9250 obj=.rgw:.bucket.meta.switch-original-staging:default.4063334.16 state=0x7f2638022398 s->prefetch_data=0
2015-01-29 15:11:30.138146 7f25616fa700 10 cache get: name=.rgw+.bucket.meta.switch-original-staging:default.4063334.16 : hit
2015-01-29 15:11:30.138150 7f25616fa700 20 get_obj_state: s->obj_tag was set empty
2015-01-29 15:11:30.138158 7f25616fa700 20 Read xattr: user.rgw.acl
2015-01-29 15:11:30.138158 7f25616fa700 20 Read xattr: user.rgw.cors
2015-01-29 15:11:30.138159 7f25616fa700 20 Read xattr: user.rgw.idtag
2015-01-29 15:11:30.138160 7f25616fa700 20 Read xattr: user.rgw.manifest
2015-01-29 15:11:30.138162 7f25616fa700 10 cache get: name=.rgw+.bucket.meta.switch-original-staging:default.4063334.16 : hit
2015-01-29 15:11:30.138175 7f25616fa700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>f2867c1c7dbd4b3b95ee616965ebabc0</ID><DisplayName>SWITCHtube-staging</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>f2867c1c7dbd4b3b95ee616965ebabc0</ID><DisplayName>SWITCHtube-staging</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-01-29 15:11:30.138182 7f25616fa700  2 req 1123:0.000190:s3:POST /:post_obj:init op
2015-01-29 15:11:30.138184 7f25616fa700  2 req 1123:0.000193:s3:POST /:post_obj:verifying op mask
2015-01-29 15:11:30.138186 7f25616fa700 20 required_mask= 2 user.op_mask=7
2015-01-29 15:11:30.138187 7f25616fa700  2 req 1123:0.000196:s3:POST /:post_obj:verifying op permissions
2015-01-29 15:11:30.138188 7f25616fa700  2 req 1123:0.000197:s3:POST /:post_obj:verifying op params
2015-01-29 15:11:30.138190 7f25616fa700  2 req 1123:0.000199:s3:POST /:post_obj:executing
2015-01-29 15:11:30.138233 7f25616fa700 20 request content_type_str=multipart/form-data; boundary=----WebKitFormBoundary1l6shWgvNaxvykeY
2015-01-29 15:11:30.138235 7f25616fa700 20 request content_type params:
2015-01-29 15:11:30.138236 7f25616fa700 20  boundary -> ----WebKitFormBoundary1l6shWgvNaxvykeY
2015-01-29 15:11:30.138237 7f25616fa700 20 adding bucket to policy env: switch-original-staging
2015-01-29 15:11:30.148502 7f25616fa700 20 read part header: name=AWSAccessKeyId content_type=
2015-01-29 15:11:30.148515 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.148516 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.148517 7f25616fa700 20 params:
2015-01-29 15:11:30.148518 7f25616fa700 20  name -> AWSAccessKeyId
2015-01-29 15:11:30.148647 7f25616fa700 20 read part header: name=key content_type=
2015-01-29 15:11:30.148649 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.148650 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.148651 7f25616fa700 20 params:
2015-01-29 15:11:30.148652 7f25616fa700 20  name -> key
2015-01-29 15:11:30.148775 7f25616fa700 20 read part header: name=acl content_type=
2015-01-29 15:11:30.148777 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.148778 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.148779 7f25616fa700 20 params:
2015-01-29 15:11:30.148780 7f25616fa700 20  name -> acl
2015-01-29 15:11:30.148902 7f25616fa700 20 read part header: name=success_action_redirect content_type=
2015-01-29 15:11:30.148904 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.148905 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.148906 7f25616fa700 20 params:
2015-01-29 15:11:30.148906 7f25616fa700 20  name -> success_action_redirect
2015-01-29 15:11:30.149022 7f25616fa700 20 read part header: name=policy content_type=
2015-01-29 15:11:30.149024 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.149025 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.149026 7f25616fa700 20 params:
2015-01-29 15:11:30.149027 7f25616fa700 20  name -> policy
2015-01-29 15:11:30.149145 7f25616fa700 20 read part header: name=signature content_type=
2015-01-29 15:11:30.149147 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.149148 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.149148 7f25616fa700 20 params:
2015-01-29 15:11:30.149149 7f25616fa700 20  name -> signature
2015-01-29 15:11:30.149438 7f25616fa700 20 read part header: name=file content_type=
2015-01-29 15:11:30.149441 7f25616fa700 20 name=Content-Disposition
2015-01-29 15:11:30.149442 7f25616fa700 20 val=form-data
2015-01-29 15:11:30.149442 7f25616fa700 20 params:
2015-01-29 15:11:30.149444 7f25616fa700 20  filename -> h264-720p.mp4
2015-01-29 15:11:30.149445 7f25616fa700 20  name -> file
2015-01-29 15:11:30.149446 7f25616fa700 20 read part header: name=file content_type=
2015-01-29 15:11:30.149447 7f25616fa700 20 name=Content-Type
2015-01-29 15:11:30.149447 7f25616fa700 20 val=video/mp4
2015-01-29 15:11:30.149448 7f25616fa700 20 params:
2015-01-29 15:11:30.149483 7f25616fa700 20 get_obj_state: rctx=0x7f263840b620 obj=.users:bd8aaf1e3ab84d329999484905572c34 state=0x7f263800dd08 s->prefetch_data=0
2015-01-29 15:11:30.149499 7f25616fa700 10 cache get: name=.users+bd8aaf1e3ab84d329999484905572c34 : miss
2015-01-29 15:11:30.151142 7f25616fa700 10 cache put: name=.users+bd8aaf1e3ab84d329999484905572c34
2015-01-29 15:11:30.151152 7f25616fa700 10 adding .users+bd8aaf1e3ab84d329999484905572c34 to cache LRU end
2015-01-29 15:11:30.151157 7f25616fa700  0 User lookup failed!
2015-01-29 15:11:30.151171 7f25616fa700 15 Read RGWCORSConfiguration<CORSConfiguration><CORSRule><AllowedMethod>POST</AllowedMethod><AllowedOrigin>https://staging.tube.switch.ch</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
2015-01-29 15:11:30.151184 7f25616fa700 10 Method POST is supported
2015-01-29 15:11:30.151195 7f25616fa700  2 req 1123:0.013204:s3:POST /:post_obj:http status=403
2015-01-29 15:11:30.151442 7f25616fa700  1 ====== req done req=0x7f26040c58d0 http_status=403 ======
2015-01-29 15:11:30.151456 7f25616fa700 20 process_request() returned -13
2015-01-29 15:11:32.022911 7f2668ff9700  2 RGWDataChangesLog::ChangesRenewThread: start

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3643 bytes --]

Re: radosgw + s3 + keystone + Browser-Based POST bug

[Abhishek L]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

Hi

Valery Tschopp writes:

> Hi guys,
>
> We have integrated our radosgw (v0.80.7) with our OpenStack Keystone 
> server (icehouse) successfully.
>
> The "normal" S3 operations can be executed with the Keystone user's EC2 
> credentials (EC2_ACCESS_KEY, EC2_SECRET_KEY). The radosgw correctly 
> handles these user credentials, ask keystone to validate them, and the 
> resulting objects belong to the Keystone tenant/project or the user 
> (user is member of the tenant/project).
>
> But for the "Browser-based upload POST" [1] it doesn't work! The user is 
> not correctly resolved, and the radosgw returns a 403 code!
>
> It looks like the s3 keystone integration doesn't work correctly when a 
> S3 browser-based upload POST is used.
>
> See the attached log file (radosgw.log), you can clearly see the user 
> lookup failing, and the status being set to 403:
>
>
> 2015-01-29 15:11:30.151157 7f25616fa700  0 User lookup failed!
> 2015-01-29 15:11:30.151171 7f25616fa700 15 Read 
> RGWCORSConfiguration<CORSConfiguration><CORSRule><AllowedMethod>POST</AllowedMethod><AllowedOrigin>https://staging.tube.switch.ch</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
> 2015-01-29 15:11:30.151184 7f25616fa700 10 Method POST is supported
> 2015-01-29 15:11:30.151195 7f25616fa700  2 req 1123:0.013204:s3:POST 
> /:post_obj:http status=403
>
>
> Is this a bug? Or did we miss something else?

Looks like you may be hitting http://tracker.ceph.com/issues/10062,
where s3 POST requests were failing with keystone. There is a patch that
is merged in master[1] that addresses this. We would also love
to see this ported back to firefly/giant.

[1] https://github.com/ceph/ceph/pull/3251


Regards 
--
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: radosgw + s3 + keystone + Browser-Based POST bug

[Yehuda Sadeh]

On Thu, Jan 29, 2015 at 9:09 AM, Abhishek L
<abhishek.lekshmanan@gmail.com> wrote:
> Hi
>
> Valery Tschopp writes:
>
>> Hi guys,
>>
>> We have integrated our radosgw (v0.80.7) with our OpenStack Keystone
>> server (icehouse) successfully.
>>
>> The "normal" S3 operations can be executed with the Keystone user's EC2
>> credentials (EC2_ACCESS_KEY, EC2_SECRET_KEY). The radosgw correctly
>> handles these user credentials, ask keystone to validate them, and the
>> resulting objects belong to the Keystone tenant/project or the user
>> (user is member of the tenant/project).
>>
>> But for the "Browser-based upload POST" [1] it doesn't work! The user is
>> not correctly resolved, and the radosgw returns a 403 code!
>>
>> It looks like the s3 keystone integration doesn't work correctly when a
>> S3 browser-based upload POST is used.
>>
>> See the attached log file (radosgw.log), you can clearly see the user
>> lookup failing, and the status being set to 403:
>>
>>
>> 2015-01-29 15:11:30.151157 7f25616fa700  0 User lookup failed!
>> 2015-01-29 15:11:30.151171 7f25616fa700 15 Read
>> RGWCORSConfiguration<CORSConfiguration><CORSRule><AllowedMethod>POST</AllowedMethod><AllowedOrigin>https://staging.tube.switch.ch</AllowedOrigin><AllowedHeader>*</AllowedHeader></CORSRule></CORSConfiguration>
>> 2015-01-29 15:11:30.151184 7f25616fa700 10 Method POST is supported
>> 2015-01-29 15:11:30.151195 7f25616fa700  2 req 1123:0.013204:s3:POST
>> /:post_obj:http status=403
>>
>>
>> Is this a bug? Or did we miss something else?
>
> Looks like you may be hitting http://tracker.ceph.com/issues/10062,
> where s3 POST requests were failing with keystone. There is a patch that
> is merged in master[1] that addresses this. We would also love
> to see this ported back to firefly/giant.

I just set it to get backported for firefly and giant.

Thanks,
Yehuda

>
> [1] https://github.com/ceph/ceph/pull/3251
>
>
> Regards
> --
> Abhishek

Re: radosgw + s3 + keystone + Browser-Based POST bug

[Abhishek L]

[-- Attachment #1: Type: text/plain, Size: 156 bytes --]


Yehuda Sadeh writes:
[..]
>
> I just set it to get backported for firefly and giant.
>
> Thanks,
> Yehuda
>

Thanks!!

Regards
-- 
Abhishek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: radosgw + s3 + keystone + Browser-Based POST bug

[Valery Tschopp]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On 29/01/15 18:28 , Yehuda Sadeh wrote:
[...]
>> Looks like you may be hitting http://tracker.ceph.com/issues/10062,
>> where s3 POST requests were failing with keystone. There is a patch that
>> is merged in master[1] that addresses this. We would also love
>> to see this ported back to firefly/giant.
>
> I just set it to get backported for firefly and giant.
>
> Thanks,
> Yehuda

Thanks a lot.

Do you have an ETA for the backport in firefly to be released?

Cheers,
Valery


>>
>> [1] https://github.com/ceph/ceph/pull/3251
>>
>>
>> Regards
>> --
>> Abhishek

-- 
SWITCH
--------------------------
Valery Tschopp, Software Engineer, Peta Solutions
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
email: valery.tschopp@switch.ch phone: +41 44 268 1544



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3643 bytes --]