'zombie snapshot' problem

'zombie snapshot' problem

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Hi,

Somehow I have managed to produce unkillable snapshot, which does not
allow to remove itself or parent image:

$ rbd snap purge dev-rack0/vm2
Removing all snapshots: 100% complete...done.
$ rbd rm dev-rack0/vm2
2012-11-21 16:31:24.184626 7f7e0d172780 -1 librbd: image has snapshots
- not removing
Removing image: 0% complete...failed.
rbd: image has snapshots - these must be deleted with 'rbd snap purge'
before the image can be removed.
$ rbd snap ls dev-rack0/vm2
SNAPID NAME                               SIZE
   188 vm2.snap-yxf 16384 MB
$ rbd info dev-rack0/vm2
rbd image 'vm2':
        size 16384 MB in 4096 objects
        order 22 (4096 KB objects)
        block_name_prefix: rbd_data.1fa164c960874
        format: 2
        features: layering
$ rbd snap rm --snap vm2.snap-yxf dev-rack0/vm2
rbd: failed to remove snapshot: (2) No such file or directory
$ rbd snap create --snap vm2.snap-yxf dev-rack0/vm2
rbd: failed to create snapshot: (17) File exists
$ rbd snap rollback --snap vm2.snap-yxf dev-rack0/vm2
Rolling back to snapshot: 100% complete...done.
$ rbd snap protect --snap vm2.snap-yxf dev-rack0/vm2
$ rbd snap unprotect --snap vm2.snap-yxf dev-rack0/vm2


Meanwhile, ``rbd ls -l dev-rack0''  segfaulting with an attached log.
Is there any reliable way to kill problematic snap?

[-- Attachment #2: log-crash.txt.gz --]
[-- Type: application/x-gzip, Size: 5358 bytes --]

Re: 'zombie snapshot' problem

[Josh Durgin]

On 11/21/2012 04:50 AM, Andrey Korolyov wrote:
> Hi,
>
> Somehow I have managed to produce unkillable snapshot, which does not
> allow to remove itself or parent image:
>
> $ rbd snap purge dev-rack0/vm2
> Removing all snapshots: 100% complete...done.

I see one bug with 'snap purge' ignoring the return code when removing
snaps. I just fixed this in the next branch. It's probably getting the
same error as 'rbd snap rm' below.

Could you post the output of:

     rbd snap purge dev-rack0/vm2 --debug-ms 1 --debug-rbd 20

> $ rbd rm dev-rack0/vm2
> 2012-11-21 16:31:24.184626 7f7e0d172780 -1 librbd: image has snapshots
> - not removing
> Removing image: 0% complete...failed.
> rbd: image has snapshots - these must be deleted with 'rbd snap purge'
> before the image can be removed.
> $ rbd snap ls dev-rack0/vm2
> SNAPID NAME                               SIZE
>     188 vm2.snap-yxf 16384 MB
> $ rbd info dev-rack0/vm2
> rbd image 'vm2':
>          size 16384 MB in 4096 objects
>          order 22 (4096 KB objects)
>          block_name_prefix: rbd_data.1fa164c960874
>          format: 2
>          features: layering
> $ rbd snap rm --snap vm2.snap-yxf dev-rack0/vm2
> rbd: failed to remove snapshot: (2) No such file or directory
> $ rbd snap create --snap vm2.snap-yxf dev-rack0/vm2
> rbd: failed to create snapshot: (17) File exists
> $ rbd snap rollback --snap vm2.snap-yxf dev-rack0/vm2
> Rolling back to snapshot: 100% complete...done.
> $ rbd snap protect --snap vm2.snap-yxf dev-rack0/vm2
> $ rbd snap unprotect --snap vm2.snap-yxf dev-rack0/vm2
>
>
> Meanwhile, ``rbd ls -l dev-rack0''  segfaulting with an attached log.
> Is there any reliable way to kill problematic snap?

 From this log it looks like vm2 used to be a clone, and the snapshot
vm2.snap-yxf was taken before it was flattened. Later, the parent of
vm2.snap-yxf was deleted. Is this correct?

It was a bug in 0.53 that protected snapshots could be deleted.

Josh

Re: 'zombie snapshot' problem

[Andrey Korolyov]

[-- Attachment #1: Type: text/plain, Size: 4046 bytes --]

On Thu, Nov 22, 2012 at 2:05 AM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 11/21/2012 04:50 AM, Andrey Korolyov wrote:
>>
>> Hi,
>>
>> Somehow I have managed to produce unkillable snapshot, which does not
>> allow to remove itself or parent image:
>>
>> $ rbd snap purge dev-rack0/vm2
>> Removing all snapshots: 100% complete...done.
>
>
> I see one bug with 'snap purge' ignoring the return code when removing
> snaps. I just fixed this in the next branch. It's probably getting the
> same error as 'rbd snap rm' below.
>
> Could you post the output of:
>
>     rbd snap purge dev-rack0/vm2 --debug-ms 1 --debug-rbd 20
>
>
>> $ rbd rm dev-rack0/vm2
>> 2012-11-21 16:31:24.184626 7f7e0d172780 -1 librbd: image has snapshots
>> - not removing
>> Removing image: 0% complete...failed.
>> rbd: image has snapshots - these must be deleted with 'rbd snap purge'
>> before the image can be removed.
>> $ rbd snap ls dev-rack0/vm2
>> SNAPID NAME                               SIZE
>>     188 vm2.snap-yxf 16384 MB
>> $ rbd info dev-rack0/vm2
>> rbd image 'vm2':
>>          size 16384 MB in 4096 objects
>>          order 22 (4096 KB objects)
>>          block_name_prefix: rbd_data.1fa164c960874
>>          format: 2
>>          features: layering
>> $ rbd snap rm --snap vm2.snap-yxf dev-rack0/vm2
>> rbd: failed to remove snapshot: (2) No such file or directory
>> $ rbd snap create --snap vm2.snap-yxf dev-rack0/vm2
>> rbd: failed to create snapshot: (17) File exists
>> $ rbd snap rollback --snap vm2.snap-yxf dev-rack0/vm2
>> Rolling back to snapshot: 100% complete...done.
>> $ rbd snap protect --snap vm2.snap-yxf dev-rack0/vm2
>> $ rbd snap unprotect --snap vm2.snap-yxf dev-rack0/vm2
>>
>>
>> Meanwhile, ``rbd ls -l dev-rack0''  segfaulting with an attached log.
>> Is there any reliable way to kill problematic snap?
>
>
> From this log it looks like vm2 used to be a clone, and the snapshot
> vm2.snap-yxf was taken before it was flattened. Later, the parent of
> vm2.snap-yxf was deleted. Is this correct?

I have attached log you asked, hope it will be useful.
Here is a two possible flows: snapshot created before and during flatten:

Completely linear flow:

$ rbd cp install/debian7 dev-rack0/testimg
Image copy: 100% complete...done.
$ rbd snap create --snap test1 dev-rack0/testimg
$ rbd snap clone --snap test1 dev-rack0/testimg dev-rack0/testimg2
rbd: error parsing command 'clone'
$ rbd snap protect --snap test1 dev-rack0/testimg
$ rbd clone --snap test1 dev-rack0/testimg dev-rack0/testimg2
$ rbd snap create --snap test2 dev-rack0/testimg2
$ rbd flatten dev-rack0/testimg2
Image flatten: 100% complete...done.
$ rbd snap unprotect --snap test1 dev-rack0/testimg
2012-11-22 15:11:03.446892 7ff9fb7c1780 -1 librbd: snap_unprotect:
can't unprotect; at least 1 child(ren) in pool dev-rack0
rbd: unprotecting snap failed: (16) Device or resource busy
$ rbd snap purge dev-rack0/testimg2
Removing all snapshots: 100% complete...done.
$ rbd snap ls dev-rack0/testimg2
$ rbd snap unprotect --snap test1 dev-rack0/testimg


snapshot created over image with ``flatten'' in progress:

$ rbd snap create --snap test3 dev-rack0/testimg
$ rbd snap protect --snap test3 dev-rack0/testimg
$ rbd clone --snap test3 dev-rack0/testimg dev-rack0/testimg3
rbd $ rbd flatten dev-rack0/testimg3
[here was executed rbd snap create --snap test43 dev-rack0/testimg3]
Image flatten: 100% complete...done.
$ rbd snap unprotect --snap test3 dev-rack0/testimg
$ rbd snap ls dev-rack0/testimg3
SNAPID NAME     SIZE
   323 test43 640 MB
$ rbd snap purge dev-rack0/testimg3
Removing all snapshots: 100% complete...done.
$ rbd snap ls dev-rack0/testimg3
SNAPID NAME     SIZE
   323 test43 640 MB
$ rbd snap rm --snap test43 dev-rack0/testimg3
rbd: failed to remove snapshot: (2) No such file or directory

Hooray, problem found! Now I`ll avoid this by putting flatten state as
exclusive one over the image.
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)

>
> It was a bug in 0.53 that protected snapshots could be deleted.
>
> Josh

[-- Attachment #2: snap.txt.gz --]
[-- Type: application/x-gzip, Size: 1685 bytes --]