Re: Latest bobtail branch still crashing KVM VMs in bh_write_commit()

[Josh Durgin <josh.durgin@inktank.com>]

I think I found the root cause based on your logs:

http://tracker.ceph.com/issues/4531

Josh

On 03/20/2013 02:47 PM, Travis Rhoden wrote:
> Didn't take long to re-create with the detailed debugging (ms =  20).
> I'm sending Josh a link to the gzip'd log off-list, I"m not sure if
> the log will contain any CephX keys or anything like that.
>
> On Wed, Mar 20, 2013 at 4:39 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>> Thanks Josh.  I will respond when I have something useful!
>>
>> On Wed, Mar 20, 2013 at 4:32 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
>>> On 03/20/2013 01:19 PM, Josh Durgin wrote:
>>>>
>>>> On 03/20/2013 01:14 PM, Stefan Priebe wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> In this case, they are format 2. And they are from cloned snapshots.
>>>>>> Exactly like the following:
>>>>>>
>>>>>> # rbd ls -l -p volumes
>>>>>> NAME                                                         SIZE
>>>>>> PARENT                                           FMT PROT LOCK
>>>>>> volume-099a6d74-05bd-4f00-a12e-009d60629aa8                 5120M
>>>>>> images/b8bdda90-664b-4906-86d6-dd33735441f2@snap   2
>>>>>>
>>>>>> I'm doing an OpenStack boot-from-volume setup.
>>>>>
>>>>>
>>>>> OK i've never used cloned snapshots so maybe this is the reason.
>>>>>
>>>>>>> strange i've never seen this. Which qemu version?
>>>>>>
>>>>>>
>>>>>> # qemu-x86_64 -version
>>>>>> qemu-x86_64 version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008
>>>>>> Fabrice Bellard
>>>>>>
>>>>>> that's coming from Ubuntu 12.04 apt repos.
>>>>>
>>>>>
>>>>> maybe you should try qemu 1.4 there are a LOT of bugfixes. qemu-kvm does
>>>>> not exist anymore it was merged into qemu with 1.3 or 1.4.
>>>>
>>>>
>>>> This particular problem won't be solved by upgrading qemu. It's a ceph
>>>> bug. Disabling caching would work around the issue.
>>>>
>>>> Travis, could you get a log from qemu of this happening with:
>>>>
>>>> debug ms = 20
>>>> debug objectcacher = 20
>>>> debug rbd = 20
>>>> log file = /path/writeable/by/qemu
>>>
>>>
>>> If it doesn't reproduce with those settings, try changing debug ms to 1
>>> instead of 20.
>>>
>>>
>>>>   From those we can tell whether the issue is on the client side at least,
>>>> and hopefully what's causing it.
>>>>
>>>> Thanks!
>>>> Josh
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>