debugging librbd to a VM

debugging librbd to a VM

[Deneau, Tom]

I have two small clusters configured with the same version of Ceph and the same OS version.
(different processor hardware)

On these clusters I have bootable RBD images that I expect a VM to boot from.
On one cluster, the VM does not boot from the image.  On the other cluster, it boots fine.
On the failing cluster, a VM can boot fine from the base qcow2 image from which the rbd image was created.  It's only rbd images that are causing problems for the VM.

I would like to enable some debug logging that could tell where the two clusters are diverging.

I tried adding
        debug rbd = 20
        log file = /tmp/cbt/ceph/log/$name.log

I do see some rbd debug messages to stderr during operations like rbd clone, etc. but I don't see any log files created in /tmp/cbt/ceph or /var/log/ceph or any messages when librbd is being used by say qemu-img convert, or by qemu-kvm.

Is there a way to see detailed debug messages for librbd?

Or is there a better way to see where things are diverging between these two clusters?
(note: I tried lttng but could not get it to record anything)

-- Tom

Re: debugging librbd to a VM

[Mykola Golub]

May be qemu does not see you ceph.conf? Or some other environmental
issue (e.g. with qemu rbd driver). Try

ldd /usr/bin/qemu-system-x86_64 |grep rbd

Have you tried starting it from command line?

You could check if config debug options work as expected setting them
on your working cluster.

On Fri, Aug 26, 2016 at 09:08:30PM +0000, Deneau, Tom wrote:
> I have two small clusters configured with the same version of Ceph and the same OS version.
> (different processor hardware)
> 
> On these clusters I have bootable RBD images that I expect a VM to boot from.
> On one cluster, the VM does not boot from the image.  On the other cluster, it boots fine.
> On the failing cluster, a VM can boot fine from the base qcow2 image from which the rbd image was created.  It's only rbd images that are causing problems for the VM.
> 
> I would like to enable some debug logging that could tell where the two clusters are diverging.
> 
> I tried adding
>         debug rbd = 20
>         log file = /tmp/cbt/ceph/log/$name.log
> 
> I do see some rbd debug messages to stderr during operations like rbd clone, etc. but I don't see any log files created in /tmp/cbt/ceph or /var/log/ceph or any messages when librbd is being used by say qemu-img convert, or by qemu-kvm.
> 
> Is there a way to see detailed debug messages for librbd?
> 
> Or is there a better way to see where things are diverging between these two clusters?
> (note: I tried lttng but could not get it to record anything)
> 
> -- Tom
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Mykola Golub

RE: debugging librbd to a VM

[Deneau, Tom]

Mykola, Jason --

I notice on both clusters the qemu process running is /usr/libexec/qemu-kvm (this is RHEL 7.2) and the euser,fgroup,ruser,uname printed by ps are all user qemu.

(note: I am initiating the VM thru virsh define xyz.xml followed by virsh start, and the user initiating both those virsh commands is root).

On both clusters /etc/ceph/ceph.conf is 644 access.

On both clusters, I get
ldd /usr/libexec/qemu-kvm | grep rbd
    librbd.so.1 => /lib64/librbd.so.1 (0x00007fbdf1071000)


When I try 
    rbd --id admin export <image-spec> - | md5sum
I get the same md5sum values on both clusters.

If I use --id qemu instead, I get 
    rbd: couldn't connect to the cluster
on both clusters.

If I leave off the --id option entirely, it works on both clusters and I get the same md5sum.

Have not tried setting debug options on the working cluster, will try that.

-- Tom


> -----Original Message-----
> From: Mykola Golub [mailto:mgolub@mirantis.com]
> Sent: Monday, August 29, 2016 3:03 AM
> To: Deneau, Tom <tom.deneau@amd.com>
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: debugging librbd to a VM
> 
> May be qemu does not see you ceph.conf? Or some other environmental
> issue (e.g. with qemu rbd driver). Try
> 
> ldd /usr/bin/qemu-system-x86_64 |grep rbd
> 
> Have you tried starting it from command line?
> 
> You could check if config debug options work as expected setting them
> on your working cluster.
> 
> On Fri, Aug 26, 2016 at 09:08:30PM +0000, Deneau, Tom wrote:
> > I have two small clusters configured with the same version of Ceph and
> the same OS version.
> > (different processor hardware)
> >
> > On these clusters I have bootable RBD images that I expect a VM to boot
> from.
> > On one cluster, the VM does not boot from the image.  On the other
> cluster, it boots fine.
> > On the failing cluster, a VM can boot fine from the base qcow2 image
> from which the rbd image was created.  It's only rbd images that are
> causing problems for the VM.
> >
> > I would like to enable some debug logging that could tell where the two
> clusters are diverging.
> >
> > I tried adding
> >         debug rbd = 20
> >         log file = /tmp/cbt/ceph/log/$name.log
> >
> > I do see some rbd debug messages to stderr during operations like rbd
> clone, etc. but I don't see any log files created in /tmp/cbt/ceph or
> /var/log/ceph or any messages when librbd is being used by say qemu-img
> convert, or by qemu-kvm.
> >
> > Is there a way to see detailed debug messages for librbd?
> >
> > Or is there a better way to see where things are diverging between these
> two clusters?
> > (note: I tried lttng but could not get it to record anything)
> >
> > -- Tom
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> Mykola Golub

Re: debugging librbd to a VM

[Brad Hubbard]

On Mon, Aug 29, 2016 at 6:03 PM, Mykola Golub <mgolub@mirantis.com> wrote:
> May be qemu does not see you ceph.conf? Or some other environmental
> issue (e.g. with qemu rbd driver). Try
>
> ldd /usr/bin/qemu-system-x86_64 |grep rbd
>
> Have you tried starting it from command line?
>
> You could check if config debug options work as expected setting them
> on your working cluster.

You can also try "touch /tmp/cbt/ceph/log/$name.log" and
"chmod 777 /tmp/cbt/ceph/log/$name.log" to eliminate permissions issues.

>
> On Fri, Aug 26, 2016 at 09:08:30PM +0000, Deneau, Tom wrote:
>> I have two small clusters configured with the same version of Ceph and the same OS version.
>> (different processor hardware)
>>
>> On these clusters I have bootable RBD images that I expect a VM to boot from.
>> On one cluster, the VM does not boot from the image.  On the other cluster, it boots fine.
>> On the failing cluster, a VM can boot fine from the base qcow2 image from which the rbd image was created.  It's only rbd images that are causing problems for the VM.
>>
>> I would like to enable some debug logging that could tell where the two clusters are diverging.
>>
>> I tried adding
>>         debug rbd = 20
>>         log file = /tmp/cbt/ceph/log/$name.log
>>
>> I do see some rbd debug messages to stderr during operations like rbd clone, etc. but I don't see any log files created in /tmp/cbt/ceph or /var/log/ceph or any messages when librbd is being used by say qemu-img convert, or by qemu-kvm.
>>
>> Is there a way to see detailed debug messages for librbd?
>>
>> Or is there a better way to see where things are diverging between these two clusters?
>> (note: I tried lttng but could not get it to record anything)
>>
>> -- Tom
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> Mykola Golub
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Cheers,
Brad

RE: debugging librbd to a VM

[Deneau, Tom]

So in my case what would $name be?

-- Tom

> -----Original Message-----
> From: Brad Hubbard [mailto:bhubbard@redhat.com]
> Sent: Monday, August 29, 2016 5:29 PM
> To: Mykola Golub <mgolub@mirantis.com>
> Cc: Deneau, Tom <tom.deneau@amd.com>; ceph-devel@vger.kernel.org
> Subject: Re: debugging librbd to a VM
> 
> On Mon, Aug 29, 2016 at 6:03 PM, Mykola Golub <mgolub@mirantis.com> wrote:
> > May be qemu does not see you ceph.conf? Or some other environmental
> > issue (e.g. with qemu rbd driver). Try
> >
> > ldd /usr/bin/qemu-system-x86_64 |grep rbd
> >
> > Have you tried starting it from command line?
> >
> > You could check if config debug options work as expected setting them
> > on your working cluster.
> 
> You can also try "touch /tmp/cbt/ceph/log/$name.log" and "chmod 777
> /tmp/cbt/ceph/log/$name.log" to eliminate permissions issues.
> 
> >
> > On Fri, Aug 26, 2016 at 09:08:30PM +0000, Deneau, Tom wrote:
> >> I have two small clusters configured with the same version of Ceph and
> the same OS version.
> >> (different processor hardware)
> >>
> >> On these clusters I have bootable RBD images that I expect a VM to boot
> from.
> >> On one cluster, the VM does not boot from the image.  On the other
> cluster, it boots fine.
> >> On the failing cluster, a VM can boot fine from the base qcow2 image
> from which the rbd image was created.  It's only rbd images that are
> causing problems for the VM.
> >>
> >> I would like to enable some debug logging that could tell where the two
> clusters are diverging.
> >>
> >> I tried adding
> >>         debug rbd = 20
> >>         log file = /tmp/cbt/ceph/log/$name.log
> >>
> >> I do see some rbd debug messages to stderr during operations like rbd
> clone, etc. but I don't see any log files created in /tmp/cbt/ceph or
> /var/log/ceph or any messages when librbd is being used by say qemu-img
> convert, or by qemu-kvm.
> >>
> >> Is there a way to see detailed debug messages for librbd?
> >>
> >> Or is there a better way to see where things are diverging between
> these two clusters?
> >> (note: I tried lttng but could not get it to record anything)
> >>
> >> -- Tom
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in the body of a message to majordomo@vger.kernel.org More majordomo
> >> info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Mykola Golub
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> --
> Cheers,
> Brad

Re: debugging librbd to a VM

[Brad Hubbard]

On Tue, Aug 30, 2016 at 8:53 AM, Deneau, Tom <tom.deneau@amd.com> wrote:
> So in my case what would $name be?

OK. To begin with just hard code a consistent name.

/var/log/ceph/rbd.log

Use that in the conf file, touch it and chmod it then restart you qemu
instance (don't
just restart the VM).

>
> -- Tom
>
>> -----Original Message-----
>> From: Brad Hubbard [mailto:bhubbard@redhat.com]
>> Sent: Monday, August 29, 2016 5:29 PM
>> To: Mykola Golub <mgolub@mirantis.com>
>> Cc: Deneau, Tom <tom.deneau@amd.com>; ceph-devel@vger.kernel.org
>> Subject: Re: debugging librbd to a VM
>>
>> On Mon, Aug 29, 2016 at 6:03 PM, Mykola Golub <mgolub@mirantis.com> wrote:
>> > May be qemu does not see you ceph.conf? Or some other environmental
>> > issue (e.g. with qemu rbd driver). Try
>> >
>> > ldd /usr/bin/qemu-system-x86_64 |grep rbd
>> >
>> > Have you tried starting it from command line?
>> >
>> > You could check if config debug options work as expected setting them
>> > on your working cluster.
>>
>> You can also try "touch /tmp/cbt/ceph/log/$name.log" and "chmod 777
>> /tmp/cbt/ceph/log/$name.log" to eliminate permissions issues.
>>
>> >
>> > On Fri, Aug 26, 2016 at 09:08:30PM +0000, Deneau, Tom wrote:
>> >> I have two small clusters configured with the same version of Ceph and
>> the same OS version.
>> >> (different processor hardware)
>> >>
>> >> On these clusters I have bootable RBD images that I expect a VM to boot
>> from.
>> >> On one cluster, the VM does not boot from the image.  On the other
>> cluster, it boots fine.
>> >> On the failing cluster, a VM can boot fine from the base qcow2 image
>> from which the rbd image was created.  It's only rbd images that are
>> causing problems for the VM.
>> >>
>> >> I would like to enable some debug logging that could tell where the two
>> clusters are diverging.
>> >>
>> >> I tried adding
>> >>         debug rbd = 20
>> >>         log file = /tmp/cbt/ceph/log/$name.log
>> >>
>> >> I do see some rbd debug messages to stderr during operations like rbd
>> clone, etc. but I don't see any log files created in /tmp/cbt/ceph or
>> /var/log/ceph or any messages when librbd is being used by say qemu-img
>> convert, or by qemu-kvm.
>> >>
>> >> Is there a way to see detailed debug messages for librbd?
>> >>
>> >> Or is there a better way to see where things are diverging between
>> these two clusters?
>> >> (note: I tried lttng but could not get it to record anything)
>> >>
>> >> -- Tom
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> >> in the body of a message to majordomo@vger.kernel.org More majordomo
>> >> info at  http://vger.kernel.org/majordomo-info.html
>> >
>> > --
>> > Mykola Golub
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> > in the body of a message to majordomo@vger.kernel.org More majordomo
>> > info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Cheers,
>> Brad



-- 
Cheers,
Brad

Re: debugging librbd to a VM

[Mykola Golub]

On Mon, Aug 29, 2016 at 06:33:28PM +0000, Deneau, Tom wrote:
 
> I notice on both clusters the qemu process running is
> /usr/libexec/qemu-kvm (this is RHEL 7.2) and the
> euser,fgroup,ruser,uname printed by ps are all user qemu.

> (note: I am initiating the VM thru virsh define xyz.xml followed by
> virsh start, and the user initiating both those virsh commands is
> root).

So I advise to look at ps output for exact qemu command line and
compare this with the working environments. If they are the same, try
to run this manually from command line (instead of virsh) to see if
there is useful output (under qemu user and root). If you don't see
the qemu process on the non-working environment (because it terminated
with failure), try to run here the command from the working env.

> On both clusters /etc/ceph/ceph.conf is 644 access.
> 
> On both clusters, I get
> ldd /usr/libexec/qemu-kvm | grep rbd
>     librbd.so.1 => /lib64/librbd.so.1 (0x00007fbdf1071000)
> 
> 
> When I try 
>     rbd --id admin export <image-spec> - | md5sum
> I get the same md5sum values on both clusters.
> 
> If I use --id qemu instead, I get 
>     rbd: couldn't connect to the cluster
> on both clusters.

Why did you try 'qemu' id? I doubt qemu sets it this way.

> 
> If I leave off the --id option entirely, it works on both clusters and I get the same md5sum.
> 
> Have not tried setting debug options on the working cluster, will
> try that.

So it does not look like rbd issue. I don't think 'debug rbd' is
useful here. You could try increasing 'debug ms' (ceph messages)
though.

-- 
Mykola Golub