upstream/firefly exporting the same snap 2 times results in different exports

upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe - Profihost AG]

Hi,

i remember there was a bug before in ceph not sure in which release
where exporting the same rbd snap multiple times results in different
raw images.

Currently running upstream/firefly and i'm seeing the same again.


# rbd export cephstor/disk-116@snap dump1
# sleep 10
# rbd export cephstor/disk-116@snap dump2

# md5sum -b dump*
b89198f118de59b3aa832db1bfddaf8f *dump1
f63ed9345ac2d5898483531e473772b1 *dump2

Can anybody help?

Greets,
Stefan

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Jason Dillaman]

Any chance that the snapshot was just created prior to the first export and you have a process actively writing to the image?

-- 

Jason Dillaman 
Red Hat 
dillaman@redhat.com 
http://www.redhat.com 


----- Original Message -----
> From: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>
> To: ceph-devel@vger.kernel.org
> Sent: Tuesday, July 21, 2015 8:29:46 AM
> Subject: upstream/firefly exporting the same snap 2 times results in different exports
> 
> Hi,
> 
> i remember there was a bug before in ceph not sure in which release
> where exporting the same rbd snap multiple times results in different
> raw images.
> 
> Currently running upstream/firefly and i'm seeing the same again.
> 
> 
> # rbd export cephstor/disk-116@snap dump1
> # sleep 10
> # rbd export cephstor/disk-116@snap dump2
> 
> # md5sum -b dump*
> b89198f118de59b3aa832db1bfddaf8f *dump1
> f63ed9345ac2d5898483531e473772b1 *dump2
> 
> Can anybody help?
> 
> Greets,
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe]

Am 21.07.2015 um 16:32 schrieb Jason Dillaman:
> Any chance that the snapshot was just created prior to the first export and you have a process actively writing to the image?
>

Sadly not. I executed those commands exactly as i've posted manually at 
bash.

I can reproduce this at 5 different ceph cluster and 500 vms each.

Stefan

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Jason Dillaman]

Does this still occur if you export the images to the console (i.e. "rbd export cephstor/disk-116@snap - > dump_file")?  

Would it be possible for you to provide logs from the two "rbd export" runs on your smallest VM image?  If so, please add the following to the "[client]" section of your ceph.conf:

  log file = /valid/path/to/logs/$name.$pid.log
  debug rbd = 20

I opened a ticket [1] where you can attach the logs (if they aren't too large).

[1] http://tracker.ceph.com/issues/12422

-- 

Jason Dillaman 
Red Hat 
dillaman@redhat.com 
http://www.redhat.com 


----- Original Message -----
> From: "Stefan Priebe" <s.priebe@profihost.ag>
> To: "Jason Dillaman" <dillaman@redhat.com>
> Cc: ceph-devel@vger.kernel.org
> Sent: Tuesday, July 21, 2015 12:55:43 PM
> Subject: Re: upstream/firefly exporting the same snap 2 times results in different exports
> 
> 
> Am 21.07.2015 um 16:32 schrieb Jason Dillaman:
> > Any chance that the snapshot was just created prior to the first export and
> > you have a process actively writing to the image?
> >
> 
> Sadly not. I executed those commands exactly as i've posted manually at
> bash.
> 
> I can reproduce this at 5 different ceph cluster and 500 vms each.
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe]

Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
> Does this still occur if you export the images to the console (i.e. "rbd export cephstor/disk-116@snap - > dump_file")?
>
> Would it be possible for you to provide logs from the two "rbd export" runs on your smallest VM image?  If so, please add the following to the "[client]" section of your ceph.conf:
>
>    log file = /valid/path/to/logs/$name.$pid.log
>    debug rbd = 20
>
> I opened a ticket [1] where you can attach the logs (if they aren't too large).
>
> [1] http://tracker.ceph.com/issues/12422

Will post some more details to the tracker in a few hours. It seems it 
is related to using discard inside guest but not on the FS the osd is on.

Stefan

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Josh Durgin]

On 07/21/2015 12:22 PM, Stefan Priebe wrote:
>
> Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
>> Does this still occur if you export the images to the console (i.e.
>> "rbd export cephstor/disk-116@snap - > dump_file")?
>>
>> Would it be possible for you to provide logs from the two "rbd export"
>> runs on your smallest VM image?  If so, please add the following to
>> the "[client]" section of your ceph.conf:
>>
>>    log file = /valid/path/to/logs/$name.$pid.log
>>    debug rbd = 20
>>
>> I opened a ticket [1] where you can attach the logs (if they aren't
>> too large).
>>
>> [1] http://tracker.ceph.com/issues/12422
>
> Will post some more details to the tracker in a few hours. It seems it
> is related to using discard inside guest but not on the FS the osd is on.

That sounds very odd. Could you verify via 'rados listwatchers' on an
in-use rbd image's header object that there's still a watch established?

Have you increased pgs in all those clusters recently?

Josh

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe]

Am 21.07.2015 um 21:46 schrieb Josh Durgin:
> On 07/21/2015 12:22 PM, Stefan Priebe wrote:
>>
>> Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
>>> Does this still occur if you export the images to the console (i.e.
>>> "rbd export cephstor/disk-116@snap - > dump_file")?
>>>
>>> Would it be possible for you to provide logs from the two "rbd export"
>>> runs on your smallest VM image?  If so, please add the following to
>>> the "[client]" section of your ceph.conf:
>>>
>>>    log file = /valid/path/to/logs/$name.$pid.log
>>>    debug rbd = 20
>>>
>>> I opened a ticket [1] where you can attach the logs (if they aren't
>>> too large).
>>>
>>> [1] http://tracker.ceph.com/issues/12422
>>
>> Will post some more details to the tracker in a few hours. It seems it
>> is related to using discard inside guest but not on the FS the osd is on.
>
> That sounds very odd. Could you verify via 'rados listwatchers' on an
> in-use rbd image's header object that there's still a watch established?

How can i do this exactly?

> Have you increased pgs in all those clusters recently?

Yes i bumped from 2048 to 4096 as i doubled the osds.

Stefan

> Josh
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe]

So this is really this old bug?

http://tracker.ceph.com/issues/9806

Stefan
Am 21.07.2015 um 21:46 schrieb Josh Durgin:
> On 07/21/2015 12:22 PM, Stefan Priebe wrote:
>>
>> Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
>>> Does this still occur if you export the images to the console (i.e.
>>> "rbd export cephstor/disk-116@snap - > dump_file")?
>>>
>>> Would it be possible for you to provide logs from the two "rbd export"
>>> runs on your smallest VM image?  If so, please add the following to
>>> the "[client]" section of your ceph.conf:
>>>
>>>    log file = /valid/path/to/logs/$name.$pid.log
>>>    debug rbd = 20
>>>
>>> I opened a ticket [1] where you can attach the logs (if they aren't
>>> too large).
>>>
>>> [1] http://tracker.ceph.com/issues/12422
>>
>> Will post some more details to the tracker in a few hours. It seems it
>> is related to using discard inside guest but not on the FS the osd is on.
>
> That sounds very odd. Could you verify via 'rados listwatchers' on an
> in-use rbd image's header object that there's still a watch established?
>
> Have you increased pgs in all those clusters recently?
>
> Josh
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Jason Dillaman]

> > That sounds very odd. Could you verify via 'rados listwatchers' on an
> > in-use rbd image's header object that there's still a watch established?
> 
> How can i do this exactly?
> 

You need to determine the RBD header object name.  For format 1 images (default for Firefly), the image header object is named "<image name>.rbd".  For format 2 images, you can determine the header object name via "rbd info <image spec> | grep 'block_name_prefix' | sed 's/.*rbd_data\.\(.*\)/rbd_header.\1/g'".

Once you have the RBD image header object name, you can run: "rados listwatchers -p <pool name> <RBD image header name>". 

-- 

Jason Dillaman 
Red Hat 
dillaman@redhat.com 
http://www.redhat.com 

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Josh Durgin]

Yes, I'm afraid it sounds like it is. You can double check whether the
watch exists on an image by getting the id of the image from 'rbd info
$pool/$image | grep block_name_prefix':

     block_name_prefix: rbd_data.105674b0dc51

The id is the hex number there. Append that to 'rbd_header.' and you
have the header object name. Check whether it has watchers with:

     rados listwatchers -p $pool rbd_header.105674b0dc51

If that doesn't show any watchers while the image is in use by a vm,
it's #9806.

I just merged the backport for firefly, so it'll be in 0.80.11.
Sorry it took so long to get to firefly :(. We'll need to be
more vigilant about checking non-trivial backports when we're
going through all the bugs periodically.

Josh

On 07/21/2015 12:52 PM, Stefan Priebe wrote:
> So this is really this old bug?
>
> http://tracker.ceph.com/issues/9806
>
> Stefan
> Am 21.07.2015 um 21:46 schrieb Josh Durgin:
>> On 07/21/2015 12:22 PM, Stefan Priebe wrote:
>>>
>>> Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
>>>> Does this still occur if you export the images to the console (i.e.
>>>> "rbd export cephstor/disk-116@snap - > dump_file")?
>>>>
>>>> Would it be possible for you to provide logs from the two "rbd export"
>>>> runs on your smallest VM image?  If so, please add the following to
>>>> the "[client]" section of your ceph.conf:
>>>>
>>>>    log file = /valid/path/to/logs/$name.$pid.log
>>>>    debug rbd = 20
>>>>
>>>> I opened a ticket [1] where you can attach the logs (if they aren't
>>>> too large).
>>>>
>>>> [1] http://tracker.ceph.com/issues/12422
>>>
>>> Will post some more details to the tracker in a few hours. It seems it
>>> is related to using discard inside guest but not on the FS the osd is
>>> on.
>>
>> That sounds very odd. Could you verify via 'rados listwatchers' on an
>> in-use rbd image's header object that there's still a watch established?
>>
>> Have you increased pgs in all those clusters recently?
>>
>> Josh
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Stefan Priebe - Profihost AG]

Am 21.07.2015 um 22:50 schrieb Josh Durgin:
> Yes, I'm afraid it sounds like it is. You can double check whether the
> watch exists on an image by getting the id of the image from 'rbd info
> $pool/$image | grep block_name_prefix':
> 
>     block_name_prefix: rbd_data.105674b0dc51
> 
> The id is the hex number there. Append that to 'rbd_header.' and you
> have the header object name. Check whether it has watchers with:
> 
>     rados listwatchers -p $pool rbd_header.105674b0dc51
> 
> If that doesn't show any watchers while the image is in use by a vm,
> it's #9806.

Yes it does not show any watchers.

> I just merged the backport for firefly, so it'll be in 0.80.11.
> Sorry it took so long to get to firefly :(. We'll need to be
> more vigilant about checking non-trivial backports when we're
> going through all the bugs periodically.

That would be really important. I've seen that this one was already in
upstream/firefly-backports. What's the purpose of that branch?

Greets,
Stefan

> Josh
> 
> On 07/21/2015 12:52 PM, Stefan Priebe wrote:
>> So this is really this old bug?
>>
>> http://tracker.ceph.com/issues/9806
>>
>> Stefan
>> Am 21.07.2015 um 21:46 schrieb Josh Durgin:
>>> On 07/21/2015 12:22 PM, Stefan Priebe wrote:
>>>>
>>>> Am 21.07.2015 um 19:19 schrieb Jason Dillaman:
>>>>> Does this still occur if you export the images to the console (i.e.
>>>>> "rbd export cephstor/disk-116@snap - > dump_file")?
>>>>>
>>>>> Would it be possible for you to provide logs from the two "rbd export"
>>>>> runs on your smallest VM image?  If so, please add the following to
>>>>> the "[client]" section of your ceph.conf:
>>>>>
>>>>>    log file = /valid/path/to/logs/$name.$pid.log
>>>>>    debug rbd = 20
>>>>>
>>>>> I opened a ticket [1] where you can attach the logs (if they aren't
>>>>> too large).
>>>>>
>>>>> [1] http://tracker.ceph.com/issues/12422
>>>>
>>>> Will post some more details to the tracker in a few hours. It seems it
>>>> is related to using discard inside guest but not on the FS the osd is
>>>> on.
>>>
>>> That sounds very odd. Could you verify via 'rados listwatchers' on an
>>> in-use rbd image's header object that there's still a watch established?
>>>
>>> Have you increased pgs in all those clusters recently?
>>>
>>> Josh
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: upstream/firefly exporting the same snap 2 times results in different exports

[Nathan Cutler]

On 2015-07-22 09:03, Stefan Priebe - Profihost AG wrote:
> That would be really important. I've seen that this one was already in
> upstream/firefly-backports. What's the purpose of that branch?

That is where the Stable Releases and Backports team stages backports
and does integration testing on them before they are merged into the
'firefly' named branch.

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037