Query: Best way to know if a watchdog is active (kicked)

Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and  timeleft(
WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

But what if we tried to read timeleft just in time when watchdog daemon/or some
other application had kicked it. May be we read timeleft twice at the interval
of 1 sec.

Please let me know if there is any other alternative which could be a better way
to know if watchdog is active?  Or may be it would be good to implement an ioctl
WDIOC_ACTIVE?

Thanks

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Guenter Roeck]

On 08/17/2015 10:15 PM, Pratyush Anand wrote:
> Hi,
>
> I am looking for the best way to know if a watchdog has been kicked and active.
>
> I can see a way is to read timeout(WDIOC_GETTIMEOUT) and  timeleft(
> WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.
>
> But what if we tried to read timeleft just in time when watchdog daemon/or some
> other application had kicked it. May be we read timeleft twice at the interval
> of 1 sec.
>
> Please let me know if there is any other alternative which could be a better way
> to know if watchdog is active?  Or may be it would be good to implement an ioctl
> WDIOC_ACTIVE?
>

Normally the watchdog is active if the watchdog device is open, unless the
application controlling it explicitly disabled it with WDIOC_SETOPTIONS.
Therefore, the controlling application should always know the status.
A different application can not open the watchdog device, so it won't be
able to get its status using an ioctl anyway.

Why is that insufficient ?

Guenter

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

Hi Guenter,

Thanks a lot for your quick reply.

On 17/08/2015:10:39:48 PM, Guenter Roeck wrote:
> On 08/17/2015 10:15 PM, Pratyush Anand wrote:
> >Hi,
> >
> >I am looking for the best way to know if a watchdog has been kicked and active.
> >
> >I can see a way is to read timeout(WDIOC_GETTIMEOUT) and  timeleft(
> >WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.
> >
> >But what if we tried to read timeleft just in time when watchdog daemon/or some
> >other application had kicked it. May be we read timeleft twice at the interval
> >of 1 sec.
> >
> >Please let me know if there is any other alternative which could be a better way
> >to know if watchdog is active?  Or may be it would be good to implement an ioctl
> >WDIOC_ACTIVE?
> >
> 
> Normally the watchdog is active if the watchdog device is open, unless the
> application controlling it explicitly disabled it with WDIOC_SETOPTIONS.
> Therefore, the controlling application should always know the status.
> A different application can not open the watchdog device, so it won't be
> able to get its status using an ioctl anyway.

Yes, A different application can not open in parallel, but can open once the
previous application has closed it. For example this is what I see:

--------------------------------------------------------------
# cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
cat: /dev/watchdog1: Invalid argument
wdctl: write failed: Invalid argument
Device:        /dev/watchdog1
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Timeleft:      24 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          0           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0
--------------------------------------------------------------
So, cat opened it and kicked it as well. But, it could not stop it as magic
character "V" had not not received. Therefore, when wdctl opened and read
Timeleft, it was different than Timeout.

> 
> Why is that insufficient ?

Well, let me explain the use case. Consider the situation when:
-- A system has activated its watchdog to take care of software hang. So, when
software has hanged, wdt causes to reboot, else it is kicked again before
timeout.
-- The same system has also activated kdump(kdump is a method to reboot to a
minimal stable secondary kernel in case of kernel crash). Now when wdt was still
active, there was a kernel crash and system booted to a secondary stable kernel
which copies crash related data to a safe location. Since, wdt was active so
before the desired process could complete in secondary kernel, hardware rebooted.
-- So, the watchdog device need to be stoped in secondary kernel as early as
possible. Loading of driver/module itself stops a kicked device. So, if there
could be a way to know active wdt from kernel, then the two daemon (one which
manages watchdog and other which manages kdump) can play independently, and
kdump daemon can correctly program a kdump file system to load relevant watchdog
module as early as possible.
-- Current distro implementations loads all the watchdog devices driver module
in secondary kernel, which is not nice (secondary kdump kernel should be as
minimal as possible).

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Dave Young]

Hi, Pratyush

A comment from my understanding about the background..

On 08/18/15 at 12:27pm, Pratyush Anand wrote:
> Hi Guenter,
> 
> Thanks a lot for your quick reply.
> 
> On 17/08/2015:10:39:48 PM, Guenter Roeck wrote:
> > On 08/17/2015 10:15 PM, Pratyush Anand wrote:
> > >Hi,
> > >
> > >I am looking for the best way to know if a watchdog has been kicked and active.
> > >
> > >I can see a way is to read timeout(WDIOC_GETTIMEOUT) and  timeleft(
> > >WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.
> > >
> > >But what if we tried to read timeleft just in time when watchdog daemon/or some
> > >other application had kicked it. May be we read timeleft twice at the interval
> > >of 1 sec.
> > >
> > >Please let me know if there is any other alternative which could be a better way
> > >to know if watchdog is active?  Or may be it would be good to implement an ioctl
> > >WDIOC_ACTIVE?
> > >
> > 
> > Normally the watchdog is active if the watchdog device is open, unless the
> > application controlling it explicitly disabled it with WDIOC_SETOPTIONS.
> > Therefore, the controlling application should always know the status.
> > A different application can not open the watchdog device, so it won't be
> > able to get its status using an ioctl anyway.
> 
> Yes, A different application can not open in parallel, but can open once the
> previous application has closed it. For example this is what I see:
> 
> --------------------------------------------------------------
> # cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
> cat: /dev/watchdog1: Invalid argument
> wdctl: write failed: Invalid argument
> Device:        /dev/watchdog1
> Identity:      iTCO_wdt [version 0]
> Timeout:       30 seconds
> Timeleft:      24 seconds
> FLAG           DESCRIPTION               STATUS BOOT-STATUS
> KEEPALIVEPING  Keep alive ping reply          0           0
> MAGICCLOSE     Supports magic close char      0           0
> SETTIMEOUT     Set timeout (in seconds)       0           0
> --------------------------------------------------------------
> So, cat opened it and kicked it as well. But, it could not stop it as magic
> character "V" had not not received. Therefore, when wdctl opened and read
> Timeleft, it was different than Timeout.
> 
> > 
> > Why is that insufficient ?
> 
> Well, let me explain the use case. Consider the situation when:
> -- A system has activated its watchdog to take care of software hang. So, when
> software has hanged, wdt causes to reboot, else it is kicked again before
> timeout.
> -- The same system has also activated kdump(kdump is a method to reboot to a
> minimal stable secondary kernel in case of kernel crash). Now when wdt was still
> active, there was a kernel crash and system booted to a secondary stable kernel
> which copies crash related data to a safe location. Since, wdt was active so
> before the desired process could complete in secondary kernel, hardware rebooted.
> -- So, the watchdog device need to be stoped in secondary kernel as early as

Either stop it or continue kicking before timeout are fine.

> possible. Loading of driver/module itself stops a kicked device. So, if there
> could be a way to know active wdt from kernel, then the two daemon (one which
> manages watchdog and other which manages kdump) can play independently, and
> kdump daemon can correctly program a kdump file system to load relevant watchdog
> module as early as possible.

Some drivers like iTCO_wdt can stop it during module loading. But I'm not sure all
drivers work. At least under 'nowayout' mode.

So the better way (still is a best effort solution though) should be kicking it again
before timeout.

> -- Current distro implementations loads all the watchdog devices driver module
> in secondary kernel, which is not nice (secondary kdump kernel should be as
> minimal as possible).
> 
> ~Pratyush

Thanks
Dave

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

Hi Dave,

On 18/08/2015:05:13:56 PM, Dave Young wrote:
> Hi, Pratyush
> 
> A comment from my understanding about the background..

I agree with your understanding.
Thanks a lot for clarifying it more.

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Guenter Roeck]

On 08/17/2015 11:57 PM, Pratyush Anand wrote:
> Hi Guenter,
>
> Thanks a lot for your quick reply.
>
> On 17/08/2015:10:39:48 PM, Guenter Roeck wrote:
>> On 08/17/2015 10:15 PM, Pratyush Anand wrote:
>>> Hi,
>>>
>>> I am looking for the best way to know if a watchdog has been kicked and active.
>>>
>>> I can see a way is to read timeout(WDIOC_GETTIMEOUT) and  timeleft(
>>> WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.
>>>
>>> But what if we tried to read timeleft just in time when watchdog daemon/or some
>>> other application had kicked it. May be we read timeleft twice at the interval
>>> of 1 sec.
>>>
>>> Please let me know if there is any other alternative which could be a better way
>>> to know if watchdog is active?  Or may be it would be good to implement an ioctl
>>> WDIOC_ACTIVE?
>>>
>>
>> Normally the watchdog is active if the watchdog device is open, unless the
>> application controlling it explicitly disabled it with WDIOC_SETOPTIONS.
>> Therefore, the controlling application should always know the status.
>> A different application can not open the watchdog device, so it won't be
>> able to get its status using an ioctl anyway.
>
> Yes, A different application can not open in parallel, but can open once the
> previous application has closed it. For example this is what I see:
>

Just by opening the watchdog it is going to be activated. So all you end up
knowing is if the watchdog was active before, but in either case it will now
be active. That doesn't seem to be very helpful.

> --------------------------------------------------------------
> # cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
> cat: /dev/watchdog1: Invalid argument
> wdctl: write failed: Invalid argument
> Device:        /dev/watchdog1
> Identity:      iTCO_wdt [version 0]
> Timeout:       30 seconds
> Timeleft:      24 seconds
> FLAG           DESCRIPTION               STATUS BOOT-STATUS
> KEEPALIVEPING  Keep alive ping reply          0           0
> MAGICCLOSE     Supports magic close char      0           0
> SETTIMEOUT     Set timeout (in seconds)       0           0
> --------------------------------------------------------------
> So, cat opened it and kicked it as well. But, it could not stop it as magic
> character "V" had not not received. Therefore, when wdctl opened and read
> Timeleft, it was different than Timeout.
>
>>
>> Why is that insufficient ?
>
> Well, let me explain the use case. Consider the situation when:
> -- A system has activated its watchdog to take care of software hang. So, when
> software has hanged, wdt causes to reboot, else it is kicked again before
> timeout.
> -- The same system has also activated kdump(kdump is a method to reboot to a
> minimal stable secondary kernel in case of kernel crash). Now when wdt was still
> active, there was a kernel crash and system booted to a secondary stable kernel
> which copies crash related data to a safe location. Since, wdt was active so
> before the desired process could complete in secondary kernel, hardware rebooted.
> -- So, the watchdog device need to be stoped in secondary kernel as early as
> possible. Loading of driver/module itself stops a kicked device. So, if there
> could be a way to know active wdt from kernel, then the two daemon (one which
> manages watchdog and other which manages kdump) can play independently, and
> kdump daemon can correctly program a kdump file system to load relevant watchdog
> module as early as possible.
> -- Current distro implementations loads all the watchdog devices driver module
> in secondary kernel, which is not nice (secondary kdump kernel should be as
> minimal as possible).

If the watchdog was active in the original kernel, it needs to be activated
in the crashdump kernel. I don't see a way around that. I can not comment on
"loads all the watchdog device driver module in the secondary kernel".
I would think it should be known which module to load.

One possible solution to your problem might be to have some sysfs attributes
associated with watchdog devices, one of which would be its state.
That has been on my mental task list for a while, but unfortunately
I never found the time to implement it.

Guenter

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

On 18/08/2015:05:50:06 AM, Guenter Roeck wrote:
> On 08/17/2015 11:57 PM, Pratyush Anand wrote:

[...]

> 
> One possible solution to your problem might be to have some sysfs attributes
> associated with watchdog devices, one of which would be its state.
> That has been on my mental task list for a while, but unfortunately
> I never found the time to implement it.

Well, I will give an attempt to implement state as a sysfs attributes associated
with watchdog devices, having two values ACTIVE and INACTIVE. Will send the
modification for review.
We can always add latter on more attributes and more state values.

Thanks a lot for your suggestion.

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Guenter Roeck]

On 08/18/2015 06:08 AM, Pratyush Anand wrote:
> On 18/08/2015:05:50:06 AM, Guenter Roeck wrote:
>> On 08/17/2015 11:57 PM, Pratyush Anand wrote:
>
> [...]
>
>>
>> One possible solution to your problem might be to have some sysfs attributes
>> associated with watchdog devices, one of which would be its state.
>> That has been on my mental task list for a while, but unfortunately
>> I never found the time to implement it.
>
> Well, I will give an attempt to implement state as a sysfs attributes associated
> with watchdog devices, having two values ACTIVE and INACTIVE. Will send the
> modification for review.
> We can always add latter on more attributes and more state values.
>

Sounds good. We should have a 'name' attribute as well. We'll also need
'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
Not sure what else.

The attributes should be implemented as class attributes, to ensure
that they are created and removed automatically. That may require
the watchdog class to be static instead of a pointer.

Thanks,
Guenter

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

On 18/08/2015:07:23:38 AM, Guenter Roeck wrote:
> Sounds good. We should have a 'name' attribute as well. We'll also need
> 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
> Not sure what else.
> 
> The attributes should be implemented as class attributes, to ensure
> that they are created and removed automatically. That may require
> the watchdog class to be static instead of a pointer.

Should n't attributes be implemented as device attribute as they are properties
of each device. Under the class watchdog_class, we can have multiple device on a
system, and these attributes would be different for different devices. So, what
I am thinking is to create device in __watchdog_register_device using
device_create_with_groups instead of device_create. Here we can link a
attribute_group with all attributes you suggested.

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Guenter Roeck]

On Fri, Aug 21, 2015 at 02:22:17PM +0530, Pratyush Anand wrote:
> On 18/08/2015:07:23:38 AM, Guenter Roeck wrote:
> > Sounds good. We should have a 'name' attribute as well. We'll also need
> > 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
> > Not sure what else.
> > 
> > The attributes should be implemented as class attributes, to ensure
> > that they are created and removed automatically. That may require
> > the watchdog class to be static instead of a pointer.
> 
> Should n't attributes be implemented as device attribute as they are properties
> of each device. Under the class watchdog_class, we can have multiple device on a
> system, and these attributes would be different for different devices. So, what
> I am thinking is to create device in __watchdog_register_device using
> device_create_with_groups instead of device_create. Here we can link a
> attribute_group with all attributes you suggested.
> 
Class attibutes are created automatically for each device registered which is
a member of the class. They are attached to the device, not to the class.
In other words, the infrastructure creates the attributes, the driver doesn't
have to do it, and there is no race condition where the device exists but
the attributes are not yet there.

Not sure what you mean with "different". Can you elaborate ?

Guenter

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

On 21/08/2015:08:19:03 AM, Guenter Roeck wrote:
> On Fri, Aug 21, 2015 at 02:22:17PM +0530, Pratyush Anand wrote:
> > On 18/08/2015:07:23:38 AM, Guenter Roeck wrote:
> > > Sounds good. We should have a 'name' attribute as well. We'll also need
> > > 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
> > > Not sure what else.
> > > 
> > > The attributes should be implemented as class attributes, to ensure
> > > that they are created and removed automatically. That may require
> > > the watchdog class to be static instead of a pointer.
> > 
> > Should n't attributes be implemented as device attribute as they are properties
> > of each device. Under the class watchdog_class, we can have multiple device on a
> > system, and these attributes would be different for different devices. So, what
> > I am thinking is to create device in __watchdog_register_device using
> > device_create_with_groups instead of device_create. Here we can link a
> > attribute_group with all attributes you suggested.
> > 
> Class attibutes are created automatically for each device registered which is
> a member of the class. They are attached to the device, not to the class.

Not sure, if we both are referring to same thing..
So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic
struct class watchdog_class" and other than .name and .owner I added
.class_attrs and then further added one node "state" in this class_attrs.

I have two watchdog in my system, so I was expecting
/sys/class/watchdog/watchdog0/state and /sys/class/watchdog/watchdog1/state. But
I see that there is only /sys/class/watchdog/state with the above approach.

> In other words, the infrastructure creates the attributes, the driver doesn't
> have to do it, and there is no race condition where the device exists but
> the attributes are not yet there.

Yes, I agree that individual driver does not need to do anything.
watchdog_core.c and watchdog_dev.c will take care for each of the device
registered with it. I am able to do that, but not using .class_attrs,
rather using device_create_with_groups.

> 
> Not sure what you mean with "different". Can you elaborate ?

As I said above, with the "different" I meant that we should have state in each
/sys/class/watchdog/watchdogn.

~Pratyush

Re: Query: Best way to know if a watchdog is active (kicked)

[Guenter Roeck]

On Fri, Aug 21, 2015 at 10:35:37PM +0530, Pratyush Anand wrote:
> On 21/08/2015:08:19:03 AM, Guenter Roeck wrote:
> > On Fri, Aug 21, 2015 at 02:22:17PM +0530, Pratyush Anand wrote:
> > > On 18/08/2015:07:23:38 AM, Guenter Roeck wrote:
> > > > Sounds good. We should have a 'name' attribute as well. We'll also need
> > > > 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
> > > > Not sure what else.
> > > > 
> > > > The attributes should be implemented as class attributes, to ensure
> > > > that they are created and removed automatically. That may require
> > > > the watchdog class to be static instead of a pointer.
> > > 
> > > Should n't attributes be implemented as device attribute as they are properties
> > > of each device. Under the class watchdog_class, we can have multiple device on a
> > > system, and these attributes would be different for different devices. So, what
> > > I am thinking is to create device in __watchdog_register_device using
> > > device_create_with_groups instead of device_create. Here we can link a
> > > attribute_group with all attributes you suggested.
> > > 
> > Class attibutes are created automatically for each device registered which is
> > a member of the class. They are attached to the device, not to the class.
> 
> Not sure, if we both are referring to same thing..
> So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic
> struct class watchdog_class" and other than .name and .owner I added
> .class_attrs and then further added one node "state" in this class_attrs.
> 
If you use .dev_groups instead of .class_attrs it should work.

Guenter

Re: Query: Best way to know if a watchdog is active (kicked)

[Pratyush Anand]

On 21/08/2015:10:13:36 AM, Guenter Roeck wrote:
> On Fri, Aug 21, 2015 at 10:35:37PM +0530, Pratyush Anand wrote:
> > On 21/08/2015:08:19:03 AM, Guenter Roeck wrote:
> > > On Fri, Aug 21, 2015 at 02:22:17PM +0530, Pratyush Anand wrote:
> > > > On 18/08/2015:07:23:38 AM, Guenter Roeck wrote:
> > > > > Sounds good. We should have a 'name' attribute as well. We'll also need
> > > > > 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'.
> > > > > Not sure what else.
> > > > > 
> > > > > The attributes should be implemented as class attributes, to ensure
> > > > > that they are created and removed automatically. That may require
> > > > > the watchdog class to be static instead of a pointer.
> > > > 
> > > > Should n't attributes be implemented as device attribute as they are properties
> > > > of each device. Under the class watchdog_class, we can have multiple device on a
> > > > system, and these attributes would be different for different devices. So, what
> > > > I am thinking is to create device in __watchdog_register_device using
> > > > device_create_with_groups instead of device_create. Here we can link a
> > > > attribute_group with all attributes you suggested.
> > > > 
> > > Class attibutes are created automatically for each device registered which is
> > > a member of the class. They are attached to the device, not to the class.
> > 
> > Not sure, if we both are referring to same thing..
> > So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic
> > struct class watchdog_class" and other than .name and .owner I added
> > .class_attrs and then further added one node "state" in this class_attrs.
> > 
> If you use .dev_groups instead of .class_attrs it should work.

OK.. Got it..Thanks for this info.  I see another issue with that. I will have
to make watchdog_class as global. Let me send my RFC patch, which will be very
soon and then may be we can see if some improvement can be done further.

~Pratyush