rbd lock list command failure

rbd lock list command failure

[Huamin Chen]

Hi

Running "rbd lock list" inside a Docker container yields mixed results. Sometimes I can get the right results but most times I just get errors.

A good run is like this:
[root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
2015-09-09 16:11:16.812858 7f3337ff5840 -1 did not load config file, using default settings.
There is 1 exclusive lock on this image.
Locker      ID                                                     Address                
client.4333 kubelet_lock_magic_host 10.16.154.78:0/1000001 

Same commands but with error:
[root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
2015-09-09 16:11:30.430345 7f4159766840 -1 did not load config file, using default settings.
2015-09-09 16:11:30.464193 7f4159766840 -1 librbd::ImageCtx: error finding header: (108) Cannot send after transport endpoint shutdown
rbd: error opening image foo: (108) Cannot send after transport endpoint shutdown

rbd version:
[root@host server]# docker run -ti ceph/base ceph -v
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)

ceph server version:
# ceph -v
ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)

Re: rbd lock list command failure

[Josh Durgin]

On 09/09/2015 09:26 AM, Huamin Chen wrote:
> Hi
>
> Running "rbd lock list" inside a Docker container yields mixed results. Sometimes I can get the right results but most times I just get errors.
>
> A good run is like this:
> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
> 2015-09-09 16:11:16.812858 7f3337ff5840 -1 did not load config file, using default settings.
> There is 1 exclusive lock on this image.
> Locker      ID                                                     Address
> client.4333 kubelet_lock_magic_host 10.16.154.78:0/1000001
>
> Same commands but with error:
> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
> 2015-09-09 16:11:30.430345 7f4159766840 -1 did not load config file, using default settings.
> 2015-09-09 16:11:30.464193 7f4159766840 -1 librbd::ImageCtx: error finding header: (108) Cannot send after transport endpoint shutdown
> rbd: error opening image foo: (108) Cannot send after transport endpoint shutdown

Are you using blacklisting? That is the error that a blacklisted client
gets from rados.

Re: rbd lock list command failure

[Huamin Chen]

Hey Josh,

Indeed, from "ceph -w", I found my client got blacklisted the moment the "rbd lock" command was executed.

I think this might be caused by my configuration that both Ceph OSD and rbd command are running inside the containers on the same host. 

Thanks

Huamin

----- Original Message -----
From: "Josh Durgin" <josh.durgin@inktank.com>
To: "Huamin Chen" <hchen@redhat.com>, ceph-devel@vger.kernel.org
Sent: Monday, September 14, 2015 6:35:49 PM
Subject: Re: rbd lock list command failure

On 09/09/2015 09:26 AM, Huamin Chen wrote:
> Hi
>
> Running "rbd lock list" inside a Docker container yields mixed results. Sometimes I can get the right results but most times I just get errors.
>
> A good run is like this:
> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
> 2015-09-09 16:11:16.812858 7f3337ff5840 -1 did not load config file, using default settings.
> There is 1 exclusive lock on this image.
> Locker      ID                                                     Address
> client.4333 kubelet_lock_magic_host 10.16.154.78:0/1000001
>
> Same commands but with error:
> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
> 2015-09-09 16:11:30.430345 7f4159766840 -1 did not load config file, using default settings.
> 2015-09-09 16:11:30.464193 7f4159766840 -1 librbd::ImageCtx: error finding header: (108) Cannot send after transport endpoint shutdown
> rbd: error opening image foo: (108) Cannot send after transport endpoint shutdown

Are you using blacklisting? That is the error that a blacklisted client
gets from rados.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd lock list command failure

[Josh Durgin]

On 09/15/2015 06:10 AM, Huamin Chen wrote:
> Hey Josh,
>
> Indeed, from "ceph -w", I found my client got blacklisted the moment the "rbd lock" command was executed.
>
> I think this might be caused by my configuration that both Ceph OSD and rbd command are running inside the containers on the same host.

Yeah, that's probably due to this bug: http://tracker.ceph.com/issues/13032

A previously blacklisted client might end up with the same client
address as a future process due to running in containers, and librados
using the pid as part of that address.

Josh

>
> ----- Original Message -----
> From: "Josh Durgin" <josh.durgin@inktank.com>
> To: "Huamin Chen" <hchen@redhat.com>, ceph-devel@vger.kernel.org
> Sent: Monday, September 14, 2015 6:35:49 PM
> Subject: Re: rbd lock list command failure
>
> On 09/09/2015 09:26 AM, Huamin Chen wrote:
>> Hi
>>
>> Running "rbd lock list" inside a Docker container yields mixed results. Sometimes I can get the right results but most times I just get errors.
>>
>> A good run is like this:
>> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
>> 2015-09-09 16:11:16.812858 7f3337ff5840 -1 did not load config file, using default settings.
>> There is 1 exclusive lock on this image.
>> Locker      ID                                                     Address
>> client.4333 kubelet_lock_magic_host 10.16.154.78:0/1000001
>>
>> Same commands but with error:
>> [root@host server]# docker run --privileged --net=host -v /dev:/dev -v /sys:/sys ceph/base rbd lock list foo --pool kube --id kube -m host:6789 --key=AQAMgXhVwBCeDhAA9nlPaFyfUSatGD4drFWDvQ==
>> 2015-09-09 16:11:30.430345 7f4159766840 -1 did not load config file, using default settings.
>> 2015-09-09 16:11:30.464193 7f4159766840 -1 librbd::ImageCtx: error finding header: (108) Cannot send after transport endpoint shutdown
>> rbd: error opening image foo: (108) Cannot send after transport endpoint shutdown
>
> Are you using blacklisting? That is the error that a blacklisted client
> gets from rados.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>