udev events on a disk, at boot time

udev events on a disk, at boot time

[Loic Dachary]

Hi Ilya,

On Debian Jessie 8.5 which has

root@000000000000:/home/ubuntu# uname -a
Linux 000000000000.teuthology 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux
root@000000000000:/home/ubuntu# udevadm --version
215

it looks like there are no udev ADD events at boot time and the OSD do not activate. After booting, if the disk is removed and re-added (via openstack server volume add / remove for instance), the OSD comes up as expected, which confirms the udev rules and the OSD are good.

This can be worked around by running ceph-disk activate-all which scans all disks when run from /etc/init/ceph-osd-all-starter.conf at boot time. The workaround requires that /dev/disk/by-parttypeuuid is populated[1] and this can be done by installing the 60-ceph-partuuid-workaround.rules [2] file.

Before trying to workaround the problem, I'd like to understand what is the difference with CentOS 7.2 which has

# uname -a
Linux target167114254243.teuthology 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
# udevadm --version
219

and correctly fires the ADD events at boot time.

A CentOS 7(minor version unknown) user reported the same problem as Debian Jessie 8.5 but I don't have information about the udev version nor the kernel version. I suspect this behavior has more to do with the udev + kernel version combination than with the distribution.

I've not actually traced the udev events at boot time because I'm not sure how to do that. Hopefully you have an idea about why this is happening. Otherwise I'll try to collect hard evidence.

Cheers

[1] https://github.com/ceph/ceph/blob/master/src/ceph-disk/ceph_disk/main.py#L3737
[2] https://github.com/ceph/ceph/blob/infernalis/udev/60-ceph-partuuid-workaround.rules

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udev events on a disk, at boot time

[Ruben Kerkhof]

Dear Loic,

On Tue, Jun 21, 2016 at 4:35 PM, Loic Dachary <loic@dachary.org> wrote:
> Hi Ilya,
>
> On Debian Jessie 8.5 which has
>
> root@000000000000:/home/ubuntu# uname -a
> Linux 000000000000.teuthology 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux
> root@000000000000:/home/ubuntu# udevadm --version
> 215
>
> it looks like there are no udev ADD events at boot time and the OSD do not activate. After booting, if the disk is removed and re-added (via openstack server volume add / remove for instance), the OSD comes up as expected, which confirms the udev rules and the OSD are good.
>
> This can be worked around by running ceph-disk activate-all which scans all disks when run from /etc/init/ceph-osd-all-starter.conf at boot time. The workaround requires that /dev/disk/by-parttypeuuid is populated[1] and this can be done by installing the 60-ceph-partuuid-workaround.rules [2] file.
>
> Before trying to workaround the problem, I'd like to understand what is the difference with CentOS 7.2 which has
>
> # uname -a
> Linux target167114254243.teuthology 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> # udevadm --version
> 219
>
> and correctly fires the ADD events at boot time.
>
> A CentOS 7(minor version unknown) user reported the same problem as Debian Jessie 8.5 but I don't have information about the udev version nor the kernel version. I suspect this behavior has more to do with the udev + kernel version combination than with the distribution.
>
> I've not actually traced the udev events at boot time because I'm not sure how to do that. Hopefully you have an idea about why this is happening. Otherwise I'll try to collect hard evidence.

I think I'm seeing the exact same issue as you describe on CentOS 7.2,
so this might not be limited to Debian Jessie.
After a reboot the OSDs do not activate.
With udev_log="debug" in /etc/udev/udev.conf, the journal doesn't show
any sign that /usr/lib/udev/rules.d/95-ceph-osd.rules is executed.

ruben@hlm1-pod12-ceph03: ~$ udevadm --version
219

ruben@hlm1-pod12-ceph03: ~$ uname -a
Linux hlm1-pod12-ceph03 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12
11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Let me know if there's anything I can do to help debug this.

Kind regards,

Ruben Kerkhof

Re: udev events on a disk, at boot time

[Loic Dachary]

On 21/06/2016 17:30, Ruben Kerkhof wrote:
> Dear Loic,
> 
> On Tue, Jun 21, 2016 at 4:35 PM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Ilya,
>>
>> On Debian Jessie 8.5 which has
>>
>> root@000000000000:/home/ubuntu# uname -a
>> Linux 000000000000.teuthology 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux
>> root@000000000000:/home/ubuntu# udevadm --version
>> 215
>>
>> it looks like there are no udev ADD events at boot time and the OSD do not activate. After booting, if the disk is removed and re-added (via openstack server volume add / remove for instance), the OSD comes up as expected, which confirms the udev rules and the OSD are good.
>>
>> This can be worked around by running ceph-disk activate-all which scans all disks when run from /etc/init/ceph-osd-all-starter.conf at boot time. The workaround requires that /dev/disk/by-parttypeuuid is populated[1] and this can be done by installing the 60-ceph-partuuid-workaround.rules [2] file.
>>
>> Before trying to workaround the problem, I'd like to understand what is the difference with CentOS 7.2 which has
>>
>> # uname -a
>> Linux target167114254243.teuthology 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>> # udevadm --version
>> 219
>>
>> and correctly fires the ADD events at boot time.
>>
>> A CentOS 7(minor version unknown) user reported the same problem as Debian Jessie 8.5 but I don't have information about the udev version nor the kernel version. I suspect this behavior has more to do with the udev + kernel version combination than with the distribution.
>>
>> I've not actually traced the udev events at boot time because I'm not sure how to do that. Hopefully you have an idea about why this is happening. Otherwise I'll try to collect hard evidence.
> 
> I think I'm seeing the exact same issue as you describe on CentOS 7.2,
> so this might not be limited to Debian Jessie.
> After a reboot the OSDs do not activate.
> With udev_log="debug" in /etc/udev/udev.conf, the journal doesn't show
> any sign that /usr/lib/udev/rules.d/95-ceph-osd.rules is executed.
> 
> ruben@hlm1-pod12-ceph03: ~$ udevadm --version
> 219
> 
> ruben@hlm1-pod12-ceph03: ~$ uname -a
> Linux hlm1-pod12-ceph03 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12
> 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> 
> Let me know if there's anything I can do to help debug this.

I wonder how my CentOS 7.2 setup is different. I'll try to reproduce the problem on CentOS 7.2 as well.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udev events on a disk, at boot time

[Alexandre DERUMIER]

Hi Loic,

I'll try to debug on my side.

(others proxmox users have also reported the problem with debian jessie too, so it's not related to my hardware)

----- Mail original -----
De: "Loic Dachary" <loic@dachary.org>
À: "Ruben Kerkhof" <ruben@rubenkerkhof.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>
Envoyé: Mardi 21 Juin 2016 18:39:29
Objet: Re: udev events on a disk, at boot time

On 21/06/2016 17:30, Ruben Kerkhof wrote: 
> Dear Loic, 
> 
> On Tue, Jun 21, 2016 at 4:35 PM, Loic Dachary <loic@dachary.org> wrote: 
>> Hi Ilya, 
>> 
>> On Debian Jessie 8.5 which has 
>> 
>> root@000000000000:/home/ubuntu# uname -a 
>> Linux 000000000000.teuthology 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux 
>> root@000000000000:/home/ubuntu# udevadm --version 
>> 215 
>> 
>> it looks like there are no udev ADD events at boot time and the OSD do not activate. After booting, if the disk is removed and re-added (via openstack server volume add / remove for instance), the OSD comes up as expected, which confirms the udev rules and the OSD are good. 
>> 
>> This can be worked around by running ceph-disk activate-all which scans all disks when run from /etc/init/ceph-osd-all-starter.conf at boot time. The workaround requires that /dev/disk/by-parttypeuuid is populated[1] and this can be done by installing the 60-ceph-partuuid-workaround.rules [2] file. 
>> 
>> Before trying to workaround the problem, I'd like to understand what is the difference with CentOS 7.2 which has 
>> 
>> # uname -a 
>> Linux target167114254243.teuthology 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux 
>> # udevadm --version 
>> 219 
>> 
>> and correctly fires the ADD events at boot time. 
>> 
>> A CentOS 7(minor version unknown) user reported the same problem as Debian Jessie 8.5 but I don't have information about the udev version nor the kernel version. I suspect this behavior has more to do with the udev + kernel version combination than with the distribution. 
>> 
>> I've not actually traced the udev events at boot time because I'm not sure how to do that. Hopefully you have an idea about why this is happening. Otherwise I'll try to collect hard evidence. 
> 
> I think I'm seeing the exact same issue as you describe on CentOS 7.2, 
> so this might not be limited to Debian Jessie. 
> After a reboot the OSDs do not activate. 
> With udev_log="debug" in /etc/udev/udev.conf, the journal doesn't show 
> any sign that /usr/lib/udev/rules.d/95-ceph-osd.rules is executed. 
> 
> ruben@hlm1-pod12-ceph03: ~$ udevadm --version 
> 219 
> 
> ruben@hlm1-pod12-ceph03: ~$ uname -a 
> Linux hlm1-pod12-ceph03 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 
> 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux 
> 
> Let me know if there's anything I can do to help debug this. 

I wonder how my CentOS 7.2 setup is different. I'll try to reproduce the problem on CentOS 7.2 as well. 

Cheers 

-- 
Loïc Dachary, Artisan Logiciel Libre 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udev events on a disk, at boot time

[Ruben Kerkhof]

> I wonder how my CentOS 7.2 setup is different. I'll try to reproduce the problem on CentOS 7.2 as well.
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre

Meanwhile I've tried a few different things. Rolling back Ceph to
10.2.1 didn't help. Upgrading the kernel to 4.6 from ELrepo didn't
help either.

One thing that might be different between our setups is that my root
fs is on LVM. I'll reinstall a server just to see if that matters.

Another workaround for this that works for me is running 'udevadm
trigger -c add -s block' after boot, this mounts the ceph volumes
successfully.

Kind regards,

Ruben
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udev events on a disk, at boot time

[Loic Dachary]

On 22/06/2016 13:33, Ruben Kerkhof wrote:
>> I wonder how my CentOS 7.2 setup is different. I'll try to reproduce the problem on CentOS 7.2 as well.
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
> 
> Meanwhile I've tried a few different things. Rolling back Ceph to
> 10.2.1 didn't help. Upgrading the kernel to 4.6 from ELrepo didn't
> help either.
> 
> One thing that might be different between our setups is that my root
> fs is on LVM. I'll reinstall a server just to see if that matters.
> 
> Another workaround for this that works for me is running 'udevadm
> trigger -c add -s block' after boot, this mounts the ceph volumes
> successfully.

Thanks for the additional information. It confirms that for some reason I have ADD events in my test setup and you don't. Still digging.

> 
> Kind regards,
> 
> Ruben
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: udev events on a disk, at boot time

[Ruben Kerkhof]

On Wed, Jun 22, 2016 at 2:20 PM, Loic Dachary <loic@dachary.org> wrote:
>> One thing that might be different between our setups is that my root
>> fs is on LVM. I'll reinstall a server just to see if that matters.
>>
>> Another workaround for this that works for me is running 'udevadm
>> trigger -c add -s block' after boot, this mounts the ceph volumes
>> successfully.
>
> Thanks for the additional information. It confirms that for some reason I have ADD events in my test setup and you don't. Still digging.

I now reinstalled a few servers with the root fs on a plain partition,
to take LVM out of the equation, and the OSDs come up fine.

Re: udev events on a disk, at boot time

[Loic Dachary]

Yeah ! One more clue :-)

On 22/06/2016 17:07, Ruben Kerkhof wrote:
> On Wed, Jun 22, 2016 at 2:20 PM, Loic Dachary <loic@dachary.org> wrote:
>>> One thing that might be different between our setups is that my root
>>> fs is on LVM. I'll reinstall a server just to see if that matters.
>>>
>>> Another workaround for this that works for me is running 'udevadm
>>> trigger -c add -s block' after boot, this mounts the ceph volumes
>>> successfully.
>>
>> Thanks for the additional information. It confirms that for some reason I have ADD events in my test setup and you don't. Still digging.
> 
> I now reinstalled a few servers with the root fs on a plain partition,
> to take LVM out of the equation, and the OSDs come up fine.
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html