Problems with "shadow" objects

Problems with "shadow" objects

[Butkeev Stas]

Hello, all

I have ceph+RGW installation. And have some problems with "shadow" objects.

For example:
#rados ls -p .rgw.buckets|grep "default.4507.1"

.....
default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.1_5
default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_2
default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.6_4
default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.4_2
default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5
.....

Please give me advices and answer on my questions
1) How can I rm this "shadow" files?
2) What does the name of this "shadow" files?
example
with normal object:
# radosgw-admin object stat --bucket=dev --object="RegExp_tutorial.png"
and I receive information about this object.

with "shadow" object:
default.4507.1_ - bucket-id
radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
ERROR: failed to stat object, returned error: (2) No such file or directory
how can I determine name of this object

-- 
Best Regards,
Stanislav Butkeev
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with "shadow" objects

[Alexis KOALLA]

Hi Stas,
My comments below in the mail

Le 03/03/2015 12:04, Butkeev Stas a écrit :
> Hello, all
>
> I have ceph+RGW installation. And have some problems with "shadow" objects.
>
> For example:
> #rados ls -p .rgw.buckets|grep "default.4507.1"
>
> .....
> default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.1_5
> default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_2
> default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.6_4
> default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.4_2
> default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5
> .....
>
> Please give me advices and answer on my questions
> 1) How can I rm this "shadow" files?
Consider that "shadow  files" are rados object and are part of an rados 
object. In fact when the Radosgateway stores an object in the cluster, 
the object is split in several parts (called chunks) and the  size of 
each chunk is by default 4Mo. But you can configure this parameter...
You can get more details in the online cephdocumentation
If you want to remove these chunks  you just  need to remove the parent 
object to which they are linked.
> 2) What does the name of this "shadow" files?
> example
> with normal object:
> # radosgw-admin object stat --bucket=dev --object="RegExp_tutorial.png"
> and I receive information about this object.
>
> with "shadow" object:
> default.4507.1_ - bucket-id
> radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
> ERROR: failed to stat object, returned error: (2) No such file or directory
> how can I determine name of this object
In the command above you forgot to type the full object name 
"default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5 "
Your command should looks like this:
* #**rdosgw-admin object stat --bucket=dev 
--object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
*
You an can have more details in inkscope project in github. We have 
implemented a module that visualizes  and show a rados object is stored 
in a Ceph cluster.
Regards
Alex

*
*
> -- 
> Best Regards,
> Stanislav Butkeev
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message tomajordomo@vger.kernel.org
> More majordomo info athttp://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with "shadow" objects

[Butkeev Stas]

Hello, Alex
I have some questions on you answer
> If you want to remove these chunks  you just  need to remove the parent object to which they are linked.
How can I determine parent object?

>>  2) What does the name of this "shadow" files?
>>  example
>>  with normal object:
>>  # radosgw-admin object stat --bucket=dev --object="RegExp_tutorial.png"
>>  and I receive information about this object.
>>
>>  with "shadow" object:
>>  default.4507.1_ - bucket-id
>>  radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
>>  ERROR: failed to stat object, returned error: (2) No such file or directory
>>  how can I determine name of this object
>
> In the command above you forgot to type the full object name
> "default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5 "
> Your command should looks like this:
> * #**rdosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
> *

What does different between my command and your command?
my command
radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
your command
radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"

they are the same

If I try use full object name, I receive error
Exemple

[root@rgw-node-2 ~]# radosgw-admin object stat --bucket=test1111 --object="current_structure_Compressor.jpg"               
{ "name": "current_structure_Compressor.jpg",
  "size": 51249,
  "policy": { "acl": { "acl_user_map": [
                { "user": "admin",
                  "acl": 15}],
          "acl_group_map": [],
......

[root@rgw-node-2 ~]# radosgw-admin object stat --bucket=test1111 --object="default.4380.1_current_structure_Compressor.jpg"
ERROR: failed to stat object, returned error: (2) No such file or directory


-- 
Best Regards,
Stanislav Butkeev


04.03.2015, 12:47, "Alexis KOALLA" <alexis.koalla@orange.com>:
> Hi Stas,
> My comments below in the mail
>
> Le 03/03/2015 12:04, Butkeev Stas a écrit :
>>  Hello, all
>>
>>  I have ceph+RGW installation. And have some problems with "shadow" objects.
>>
>>  For example:
>>  #rados ls -p .rgw.buckets|grep "default.4507.1"
>>
>>  .....
>>  default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.1_5
>>  default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_2
>>  default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.6_4
>>  default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.4_2
>>  default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5
>>  .....
>>
>>  Please give me advices and answer on my questions
>>  1) How can I rm this "shadow" files?
>
> Consider that "shadow  files" are rados object and are part of an rados
> object. In fact when the Radosgateway stores an object in the cluster,
> the object is split in several parts (called chunks) and the  size of
> each chunk is by default 4Mo. But you can configure this parameter...
> You can get more details in the online cephdocumentation
> If you want to remove these chunks  you just  need to remove the parent
> object to which they are linked.
>>  2) What does the name of this "shadow" files?
>>  example
>>  with normal object:
>>  # radosgw-admin object stat --bucket=dev --object="RegExp_tutorial.png"
>>  and I receive information about this object.
>>
>>  with "shadow" object:
>>  default.4507.1_ - bucket-id
>>  radosgw-admin object stat --bucket=dev --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
>>  ERROR: failed to stat object, returned error: (2) No such file or directory
>>  how can I determine name of this object
>
> In the command above you forgot to type the full object name
> "default.4507.1__shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.3_5 "
> Your command should looks like this:
> * #**rdosgw-admin object stat --bucket=dev
> --object="_shadow_test_s3.2/2vO4WskQNBGMnC8MGaYPSLfGkhQY76U.2_7"
> *
> You an can have more details in inkscope project in github. We have
> implemented a module that visualizes  and show a rados object is stored
> in a Ceph cluster.
> Regards
> Alex
>
> *
> *
>>  --
>>  Best Regards,
>>  Stanislav Butkeev
>>  --
>>  To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>  the body of a message tomajordomo@vger.kernel.org
>>  More majordomo info athttp://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html