Watchdog drivers and device driver model

Watchdog drivers and device driver model

[Jean Delvare]

Hi Wim and all,

While investigating a customer case, it occurred to me that it is not
always possible to figure out which driver is handling /dev/watchdog.
For some drivers, /sys/dev/char/10:130 is a symbolic link
to /sys/devices/virtual/misc/watchdog, and that virtual device says
nothing about the physical device nor the device driver.

As far as I can see, this happens whenever the dev.parent field of the
watchdog_device structure isn't set before calling
watchdog_register_device(). Couldn't we make setting this field
mandatory, so that it is always possible to get to the physical device?
It seems that only a minority of watchdog drivers are doing it properly
at the moment.

Also, it seems to me that at least in some cases, the parent device that
is set isn't the right one. I am looking at the iTCO_wdt driver, it puts
the PCI device as the parent, however the driver itself is a platform
driver so it is attached to the iTCO_wdt platform device. That makes it
impossible to find which driver is handling /dev/watchdog in an
automated way: both /sys/dev/char/10:130
and /sys/class/watchdog/watchdog0 point to the PCI device, which has
driver "lpc_ich" attached, not "iTCO_wdt". One has to enter the iTCO_wdt
directory there to find the proper driver. May I suggest that dev.parent
should point to the iTCO_wdt platform device in that case?

If you need some background: my customer needs to unload the native
watchdog driver and use the IPMI watchdog instead. So they need a
reliable way to figure out who is handling /dev/watchdog so that they
can unload the right module. This is rather difficult, costly and
fragile with the way things are currently implemented.

Thanks,
-- 
Jean Delvare
SUSE L3 Support

Re: Watchdog drivers and device driver model

[Guenter Roeck]

On 03/07/2014 05:34 AM, Jean Delvare wrote:
> Hi Wim and all,
>
> While investigating a customer case, it occurred to me that it is not
> always possible to figure out which driver is handling /dev/watchdog.
> For some drivers, /sys/dev/char/10:130 is a symbolic link
> to /sys/devices/virtual/misc/watchdog, and that virtual device says
> nothing about the physical device nor the device driver.
>
> As far as I can see, this happens whenever the dev.parent field of the
> watchdog_device structure isn't set before calling
> watchdog_register_device(). Couldn't we make setting this field
> mandatory, so that it is always possible to get to the physical device?
> It seems that only a minority of watchdog drivers are doing it properly
> at the moment.
>
> Also, it seems to me that at least in some cases, the parent device that
> is set isn't the right one. I am looking at the iTCO_wdt driver, it puts
> the PCI device as the parent, however the driver itself is a platform
> driver so it is attached to the iTCO_wdt platform device. That makes it
> impossible to find which driver is handling /dev/watchdog in an
> automated way: both /sys/dev/char/10:130
> and /sys/class/watchdog/watchdog0 point to the PCI device, which has
> driver "lpc_ich" attached, not "iTCO_wdt". One has to enter the iTCO_wdt
> directory there to find the proper driver. May I suggest that dev.parent
> should point to the iTCO_wdt platform device in that case?
>
Good idea. I've had trouble identifying watchdogs myself, and it would
be great to have a reliable means to do it.

> If you need some background: my customer needs to unload the native
> watchdog driver and use the IPMI watchdog instead. So they need a
> reliable way to figure out who is handling /dev/watchdog so that they
> can unload the right module. This is rather difficult, costly and
> fragile with the way things are currently implemented.
>

Another solution in that specific case would be to convert the ipmi
watchdog driver to the watchdog API. Then it could be loaded in parallel
with other watchdog drivers. Of course, the naming problem would still
exist.

Guenter

Re: Watchdog drivers and device driver model

[Jean Delvare]

Hi Guenter,

Le Friday 07 March 2014 à 06:37 -0800, Guenter Roeck a écrit :
> On 03/07/2014 05:34 AM, Jean Delvare wrote:
> > If you need some background: my customer needs to unload the native
> > watchdog driver and use the IPMI watchdog instead. So they need a
> > reliable way to figure out who is handling /dev/watchdog so that they
> > can unload the right module. This is rather difficult, costly and
> > fragile with the way things are currently implemented.
>
> Another solution in that specific case would be to convert the ipmi
> watchdog driver to the watchdog API. Then it could be loaded in parallel
> with other watchdog drivers. Of course, the naming problem would still
> exist.

Not sure if it would help. If both the native watchdog driver and the
IPMI watchdog driver get loaded automatically via module aliases, it
will be racy. Only one driver can own /dev/watchdog at any given time,
and the use has to decide which one.

-- 
Jean Delvare
SUSE L3 Support

Re: Watchdog drivers and device driver model

[Guenter Roeck]

Hi Jean,

On Fri, Mar 07, 2014 at 04:30:46PM +0100, Jean Delvare wrote:
> Hi Guenter,
> 
> Le Friday 07 March 2014 à 06:37 -0800, Guenter Roeck a écrit :
> > On 03/07/2014 05:34 AM, Jean Delvare wrote:
> > > If you need some background: my customer needs to unload the native
> > > watchdog driver and use the IPMI watchdog instead. So they need a
> > > reliable way to figure out who is handling /dev/watchdog so that they
> > > can unload the right module. This is rather difficult, costly and
> > > fragile with the way things are currently implemented.
> >
> > Another solution in that specific case would be to convert the ipmi
> > watchdog driver to the watchdog API. Then it could be loaded in parallel
> > with other watchdog drivers. Of course, the naming problem would still
> > exist.
> 
> Not sure if it would help. If both the native watchdog driver and the
> IPMI watchdog driver get loaded automatically via module aliases, it
> will be racy. Only one driver can own /dev/watchdog at any given time,
> and the use has to decide which one.
> 
At work we are using /dev/watchdog0 and /dev/watchdog1 directly,
and don't depend on /dev/watchdog (which maps to /dev/watchdog0).
This is actually quite convenient, as it lets us use the two watchdogs
to watch each other (I've seen hard system lockups with the mei
watchdog and the iTCO watchdog on a couple of systems, so I like to
have a backup).

Sure, if you insist that the ipmi watchdog is tied to /dev/watchdog,
no matter what, that is a different issue.

Anyway, in our case it doesn't matter much which watchdog is on
which device, because we just use both. But I agree that it would
be nice to have an easy way to determine the underlying device
from user space (without having to go through the ioctl to get
the name).

Guenter

Re: Watchdog drivers and device driver model

[Jean Delvare]

Hi Guenter,

Le Friday 07 March 2014 à 09:29 -0800, Guenter Roeck a écrit :
> On Fri, Mar 07, 2014 at 04:30:46PM +0100, Jean Delvare wrote:
> > Not sure if it would help. If both the native watchdog driver and the
> > IPMI watchdog driver get loaded automatically via module aliases, it
> > will be racy. Only one driver can own /dev/watchdog at any given time,
> > and the use has to decide which one.
>
> At work we are using /dev/watchdog0 and /dev/watchdog1 directly,
> and don't depend on /dev/watchdog (which maps to /dev/watchdog0).
> This is actually quite convenient, as it lets us use the two watchdogs
> to watch each other (I've seen hard system lockups with the mei
> watchdog and the iTCO watchdog on a couple of systems, so I like to
> have a backup).

I admit I don't quite understand how watchdogs can "watch each other"?
Each watchdog is essentially watching the watchdog daemon which is
periodically writing to it, and assuming that daemon alive == system
alive, right?

Do you have one daemon writing to both watchdog device nodes? Or two
daemon instances each writing to one device node?

As I understand it, the only benefit of using multiple watchdogs, is if
the watchdog devices (or their drivers) are not 100% reliable, right?

> Sure, if you insist that the ipmi watchdog is tied to /dev/watchdog,
> no matter what, that is a different issue.

I think that's what the customer wants, yes. That being said, I seem to
recall that not so long ago (on the enterprise scale) you could only
have one watchdog driver running at once, right? Multiple watchdog
support was added with commit 45f5fed3 in kernel v3.5 if I'm not
mistaken. Only the latest SP of our latest product includes that commit.
I suppose this is the reason why our customers still stick
to /dev/watchdog and a single watchdog driver. It might change in the
future, but probably not before the next major version of our product is
released and widely deployed.

> Anyway, in our case it doesn't matter much which watchdog is on
> which device, because we just use both. But I agree that it would
> be nice to have an easy way to determine the underlying device
> from user space (without having to go through the ioctl to get
> the name).

I just checked on my own laptop with kernel 3.0.101 + lots of backports
and after boot I see 3 watchdog devices. All virtual so I have no way to
figure out which driver and physical device is behind
them. /dev/watchdog goes away if I unload iTCO_wdt, /dev/watchdog2 goes
away if I unload mei. /dev/watchdog1 is a mystery.

With kernel 3.12.12 + lots of backports, /dev/watchdog1 is owned by
iTCO_wdt (although sysfs points to driver lpc_ich, as mentioned in a
previous mail.) /dev/watchdog and /dev/watchdog0 are both owned by
mei_me, but I had to guess as they are still a virtual devices (one
misc, one watchdog.)

So there is still some work to get things in good shape. Also this
confirms that the watchdog device node <-> driver mapping is fragile and
can change from one kernel version to the next or even across reboot, so
users shouldn't assume it to be persistent.

-- 
Jean Delvare
SUSE L3 Support

Re: Watchdog drivers and device driver model

[Guenter Roeck]

On 03/08/2014 04:27 AM, Jean Delvare wrote:
> Hi Guenter,
>
> Le Friday 07 March 2014 à 09:29 -0800, Guenter Roeck a écrit :
>> On Fri, Mar 07, 2014 at 04:30:46PM +0100, Jean Delvare wrote:
>>> Not sure if it would help. If both the native watchdog driver and the
>>> IPMI watchdog driver get loaded automatically via module aliases, it
>>> will be racy. Only one driver can own /dev/watchdog at any given time,
>>> and the use has to decide which one.
>>
>> At work we are using /dev/watchdog0 and /dev/watchdog1 directly,
>> and don't depend on /dev/watchdog (which maps to /dev/watchdog0).
>> This is actually quite convenient, as it lets us use the two watchdogs
>> to watch each other (I've seen hard system lockups with the mei
>> watchdog and the iTCO watchdog on a couple of systems, so I like to
>> have a backup).
>
> I admit I don't quite understand how watchdogs can "watch each other"?
> Each watchdog is essentially watching the watchdog daemon which is
> periodically writing to it, and assuming that daemon alive == system
> alive, right?
>
Bad terminology. We use two watchdogs in parallel. If oe fails to work,
the other does (hopefully ;-)

> Do you have one daemon writing to both watchdog device nodes? Or two
> daemon instances each writing to one device node?
>
systemd uses one, the watchdog application the other.

> As I understand it, the only benefit of using multiple watchdogs, is if
> the watchdog devices (or their drivers) are not 100% reliable, right?
>
Correct.

>> Sure, if you insist that the ipmi watchdog is tied to /dev/watchdog,
>> no matter what, that is a different issue.
>
> I think that's what the customer wants, yes. That being said, I seem to
> recall that not so long ago (on the enterprise scale) you could only
> have one watchdog driver running at once, right? Multiple watchdog
> support was added with commit 45f5fed3 in kernel v3.5 if I'm not
> mistaken. Only the latest SP of our latest product includes that commit.
> I suppose this is the reason why our customers still stick
> to /dev/watchdog and a single watchdog driver. It might change in the
> future, but probably not before the next major version of our product is
> released and widely deployed.
>
Ok, makes sense, but I think it was much earlier than 3.5.

>> Anyway, in our case it doesn't matter much which watchdog is on
>> which device, because we just use both. But I agree that it would
>> be nice to have an easy way to determine the underlying device
>> from user space (without having to go through the ioctl to get
>> the name).
>
> I just checked on my own laptop with kernel 3.0.101 + lots of backports
> and after boot I see 3 watchdog devices. All virtual so I have no way to
> figure out which driver and physical device is behind
> them. /dev/watchdog goes away if I unload iTCO_wdt, /dev/watchdog2 goes
> away if I unload mei. /dev/watchdog1 is a mystery.
>
> With kernel 3.12.12 + lots of backports, /dev/watchdog1 is owned by
> iTCO_wdt (although sysfs points to driver lpc_ich, as mentioned in a
> previous mail.) /dev/watchdog and /dev/watchdog0 are both owned by
> mei_me, but I had to guess as they are still a virtual devices (one
> misc, one watchdog.)
>
> So there is still some work to get things in good shape. Also this
> confirms that the watchdog device node <-> driver mapping is fragile and
> can change from one kernel version to the next or even across reboot, so
> users shouldn't assume it to be persistent.
>
Just like pretty much everything else ;-). I absolutely agree that it
would be great to have a better means to determine both the driver
and the name of the watchdog.

Guenter