ON/OFF control of taskstats accounting data at do_exit

ON/OFF control of taskstats accounting data at do_exit

[Jay Lan]

Hi Balbir and Shailabh,

I propose we add the capability to turn ON/OFF the multicase of
taskstats accounting data at do_exit().

This would allow 'chkconfig taskstats' like of control similar
to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would
wish to turn off sending accounting data to the multicast socket.
We have seen many situations that our CSA customers need to turn
off CSA for a period of time.

I think this feature is very important to this new interface.

Thanks,
 - jay

Re: ON/OFF control of taskstats accounting data at do_exit

[Balbir Singh]

Jay Lan wrote:
> Hi Balbir and Shailabh,
> 
> I propose we add the capability to turn ON/OFF the multicase of
> taskstats accounting data at do_exit().
> 
> This would allow 'chkconfig taskstats' like of control similar
> to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would
> wish to turn off sending accounting data to the multicast socket.
> We have seen many situations that our CSA customers need to turn
> off CSA for a period of time.

If we need to turn the entire sending of stats off, then closing
the socket listening on the multicast group should turn off sending
of accounting information. Am I missing something?

> 
> I think this feature is very important to this new interface.
> 
> Thanks,
>  - jay
> 



-- 
	Regards,
	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs

Re: ON/OFF control of taskstats accounting data at do_exit

[Shailabh Nagar]

Jay Lan wrote:

>Hi Balbir and Shailabh,
>
>I propose we add the capability to turn ON/OFF the multicase of
>taskstats accounting data at do_exit().
>  
>

>This would allow 'chkconfig taskstats' like of control similar
>to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would
>wish to turn off sending accounting data to the multicast socket.
>We have seen many situations that our CSA customers need to turn
>off CSA for a period of time.
>
 

What happens to other clients listening to the data ? This sort of 
configuration option would
again be along the lines of a systemwide change prompted by needs of one 
subsystem but affecting
all others - something Andrew had recommended against doing while we 
were discussing the per-tgid
turnoff.

The existing way to solve this is for the listeners to close the socket 
and reopen again later when they
are interested.  If there are no listeners at all, data won't be sent by 
the kernel anyway. Won't that work
for you ?

--Shailabh

>I think this feature is very important to this new interface.
>
>Thanks,
> - jay
>  
>

Re: ON/OFF control of taskstats accounting data at do_exit

[Jay Lan]

Shailabh Nagar wrote:
> Jay Lan wrote:
>
>> Hi Balbir and Shailabh,
>>
>> I propose we add the capability to turn ON/OFF the multicase of
>> taskstats accounting data at do_exit().
>>  
>>
>
>> This would allow 'chkconfig taskstats' like of control similar
>> to 'chkconfig acct' for BSD accounting. Sometimes sysadmins would
>> wish to turn off sending accounting data to the multicast socket.
>> We have seen many situations that our CSA customers need to turn
>> off CSA for a period of time.
>>
>
>
> What happens to other clients listening to the data ? This sort of
> configuration option would
> again be along the lines of a systemwide change prompted by needs of
> one subsystem but affecting
> all others - something Andrew had recommended against doing while we
> were discussing the per-tgid
> turnoff.
>
> The existing way to solve this is for the listeners to close the
> socket and reopen again later when they
> are interested.  If there are no listeners at all, data won't be sent
> by the kernel anyway. Won't that work
> for you ?

I was talking about turning off system-wise taskstats data preparation and
delivery when every task exits. Sometimes customers like to do some
benchmarking and need to turn off nonessential features.

Are you saying if there is no listener, data will not be assembled and sent
by the kernel? I thought kernel would always send no matter whether there
is listener? I apologize for the noise if i made a mistake.

Regards,
 - jay

>
> --Shailabh
>

Re: ON/OFF control of taskstats accounting data at do_exit

[Shailabh Nagar]

Jay Lan wrote:
> 
> Are you saying if there is no listener, data will not be assembled and sent
> by the kernel?  I thought kernel would always send no matter whether there
> is listener? I apologize for the noise if i made a mistake.

It will be assembled by taskstats. But afaict, the sending of data by the netlink layer will
be halted within netlink_broadcast(), which is a couple of calls down the call path from
genlmsg_multicast(), which is used by taskstats uses to send data out.

Lets discuss turning off of taskstats assembly separately.


Jamal, Thomas,

If there are no listeners on a genetlink socket, do you think overhead of kernel paths that
continue to attempt sending multicast data on genetlink sockets is an issue ?


--Shailabh

Re: ON/OFF control of taskstats accounting data at do_exit

[Shailabh Nagar]

Jay Lan wrote:


> 
> I was talking about turning off system-wise taskstats data preparation and
> delivery when every task exits. Sometimes customers like to do some
> benchmarking and need to turn off nonessential features.

Lets go through the implications of turning on/off collection, assembly and delivery
of the per-task accounting data.

Collection is defined by different subsystems using taskstats.
For atleast one of these (delay accounting), turning on/off dynamically has been tried
and deemed to cause a lot of overhead (due to accumalated nature of data) and also be
prone to races. Complexity of code added did not justify the value so on/off was restricted
to a boot time decision.

Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill
the commonly used struct taskstats and relying on genetlink to do the delivery.
This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets
some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats
and taskstats_exit_send)
Doing so will have impact on
a) queries sent to the kernel by monitoring applications
b) task exit data sent by kernel to apps listening on the multicast socket

For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off.
Also, I'm assuming monitoring applications aren't aware of the turn off.

What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead
is saved (since such an error reply can be sent right away as soon as a query command is seen) but no
substantial saving on the delivery part.

For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't
we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't
expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?

Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace
apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay
accounting and other users turned off) is clear.

Thoughts ?

--Shailabh

Re: ON/OFF control of taskstats accounting data at do_exit

[Jay Lan]

Shailabh Nagar wrote:
> Jay Lan wrote:
> 
> 
> 
>>I was talking about turning off system-wise taskstats data preparation and
>>delivery when every task exits. Sometimes customers like to do some
>>benchmarking and need to turn off nonessential features.
> 
> 
> Lets go through the implications of turning on/off collection, assembly and delivery
> of the per-task accounting data.
> 
> Collection is defined by different subsystems using taskstats.
> For atleast one of these (delay accounting), turning on/off dynamically has been tried
> and deemed to cause a lot of overhead (due to accumalated nature of data) and also be
> prone to races. Complexity of code added did not justify the value so on/off was restricted
> to a boot time decision.

Agreed.

> 
> Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill
> the commonly used struct taskstats and relying on genetlink to do the delivery.
> This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets
> some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats
> and taskstats_exit_send)
> Doing so will have impact on
> a) queries sent to the kernel by monitoring applications
> b) task exit data sent by kernel to apps listening on the multicast socket
> 
> For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off.
> Also, I'm assuming monitoring applications aren't aware of the turn off.

I do not see impacts of both cases above since it would not happen.
I expect the event of turning off taskstats feature is coordidated
by the system adminstrator, so all users are notified in advance.

For that reason, i think exposing the switch at sysfs is not a good
idea. Instead, /etc/init.d/taskstats script would be right for
this purpose. At kernel side, we would need to make this possible.

What do you think?

Regards,
  - jay

> 
> What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead
> is saved (since such an error reply can be sent right away as soon as a query command is seen) but no
> substantial saving on the delivery part.
> 
> For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't
> we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't
> expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?
> 
> Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace
> apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay
> accounting and other users turned off) is clear.
> 
> Thoughts ?
> 
> --Shailabh
> 
> 
> 
> 
> 
> 
> 
> 

Re: ON/OFF control of taskstats accounting data at do_exit

[Balbir Singh]

 
> For that reason, i think exposing the switch at sysfs is not a good
> idea. Instead, /etc/init.d/taskstats script would be right for
> this purpose. At kernel side, we would need to make this possible.

When you say we need to make this possible at the kernel side, do you
want the kernel not to accumulate taskstats in the do_exit() path
(avoid calling fill_xxxx routines) if there are no listeners
on TASKSTATS_LISTEN_GROUP? As Shailabh, mentioned sending out does not
happen if there are no listeners.

> 
> What do you think?
> 
> Regards,
>  - jay
> 
-- 

	Balbir Singh,
	Linux Technology Center,
	IBM Software Labs

Re: ON/OFF control of taskstats accounting data at do_exit

[Jay Lan]

Balbir Singh wrote:
> 
>> For that reason, i think exposing the switch at sysfs is not a good
>> idea. Instead, /etc/init.d/taskstats script would be right for
>> this purpose. At kernel side, we would need to make this possible.
> 
> 
> When you say we need to make this possible at the kernel side, do you
> want the kernel not to accumulate taskstats in the do_exit() path
> (avoid calling fill_xxxx routines) if there are no listeners
> on TASKSTATS_LISTEN_GROUP? As Shailabh, mentioned sending out does not
> happen if there are no listeners.

I did not understand that when i made the request because i did not
see from taskstats code how you can control that. Shailabh later said
that it was done at netlink layer.

I do not know how deep down in netlink layer to exit the proces,
but it may turn out that the need of this feature not that importan...

Yet, we still allocate memory, fill taskstats struct, invoke netlink,
and free memory. Ideally when taskstats is turned off, we do not need
to do any of these. So, if we can fail memory allocation when taskstats
is turned off, the taskstats_exit_send() will return right away.

To answer Shailabh's discussion in a separate email, yes, i meant
the second case that all taskstats applications will be stopped or
suspended. The different between a /etc/init.d/taskstats and a
sysfs/procfs control of taskstats is that the system will not
be activated at init time. However, i admit customers can always turn
off taskstats via sysfs before starts whatever they like to do.

I do not know what is community's preference: a /etc/init.d mechnism
or a sysfs mechnism. I would be happy either way. Thanks!

Regards,
  jay


> 
>>
>> What do you think?
>>
>> Regards,
>>  - jay
>>

Re: ON/OFF control of taskstats accounting data at do_exit

[Shailabh Nagar]

Jay Lan wrote:
> Shailabh Nagar wrote:

>>
>> Assembly and delivery of data is done by the taskstats code, calling
>> subsystem-specific functions to fill
>> the commonly used struct taskstats and relying on genetlink to do the
>> delivery.
>> This can be turned on/off using a dynamic parameter such as
>> /sys/kernel/taskstats_enable which sets
>> some internal variable that is used to do early exits from relevant
>> functions (mainly taskstats_send_stats
>> and taskstats_exit_send)
>> Doing so will have impact on
>> a) queries sent to the kernel by monitoring applications
>> b) task exit data sent by kernel to apps listening on the multicast
>> socket
>>
>> For consistency, I'm assuming both a) and b) will have to be affected
>> when taskstats is turned off.
>> Also, I'm assuming monitoring applications aren't aware of the turn off.
> 
> 
> I do not see impacts of both cases above since it would not happen.
> I expect the event of turning off taskstats feature is coordidated
> by the system adminstrator, so all users are notified in advance.

Agreed that all users will be notified. Once that happens, what do you expect
these users will do to the applications they are running that
a) query per-task data periodically, or
b) monitor exit data

Either these applications will continue running (querying, listening) and
have to understand that they cannot expect more data
or they will be stopped(or suspended) from doing the monitoring/querying.

If its the latter, then we don't really need kernel changes since we only save on
a little bit of assembling data overhead.

If its the former, then yes, it will be useful to have the kernel changes.

> 
> For that reason, i think exposing the switch at sysfs is not a good
> idea. Instead, /etc/init.d/taskstats script would be right for
> this purpose. 

I don't get the difference between these two options as far as kernel is concerned.
A systemwide switch that can be set by only by a privileged user is what is needed
either way, right ? Whether it is in sys/kernel (through sysfs) or
/proc/sys/kernel (through /proc and via sysctl) ? Or is there something else that an
/etc/init.d/taskstats script can do to set a kernel parameter ?


> At kernel side, we would need to make this possible.
> 
> What do you think?
> 
> Regards,
>  - jay
> 
>>
>> What happens to case a) ? Apps will need to get some error message as
>> a reply. Some assembly overhead
>> is saved (since such an error reply can be sent right away as soon as
>> a query command is seen) but no
>> substantial saving on the delivery part.
>>
>> For case b), we can save on assembly and delivery by exiting early
>> from taskstats_exit_send(). But won't
>> we need to send some message (if not periodically, atleast once) to
>> listening apps that they shouldn't
>> expect any exit data ? Semantics of suddenly not seeing any exit data
>> could be misinterpreted ?
>>
>> Its easy enough to implement...just concerned about the semantics of
>> doing so (as far as userspace
>> apps are concerned) and utility in general settings. Utility in case
>> where only CSA is running (delay
>> accounting and other users turned off) is clear.
>>
>> Thoughts ?
>>
>> --Shailabh
>>
>>
>>
>>
>>
>>
>>
>>
> 

Re: ON/OFF control of taskstats accounting data at do_exit

[Peter Williams]

Shailabh Nagar wrote:
> Jay Lan wrote:
> 
> 
>> I was talking about turning off system-wise taskstats data preparation and
>> delivery when every task exits. Sometimes customers like to do some
>> benchmarking and need to turn off nonessential features.
> 
> Lets go through the implications of turning on/off collection, assembly and delivery
> of the per-task accounting data.
> 
> Collection is defined by different subsystems using taskstats.
> For atleast one of these (delay accounting), turning on/off dynamically has been tried
> and deemed to cause a lot of overhead (due to accumalated nature of data) and also be
> prone to races. Complexity of code added did not justify the value so on/off was restricted
> to a boot time decision.
> 
> Assembly and delivery of data is done by the taskstats code, calling subsystem-specific functions to fill
> the commonly used struct taskstats and relying on genetlink to do the delivery.
> This can be turned on/off using a dynamic parameter such as /sys/kernel/taskstats_enable which sets
> some internal variable that is used to do early exits from relevant functions (mainly taskstats_send_stats
> and taskstats_exit_send)
> Doing so will have impact on
> a) queries sent to the kernel by monitoring applications
> b) task exit data sent by kernel to apps listening on the multicast socket
> 
> For consistency, I'm assuming both a) and b) will have to be affected when taskstats is turned off.
> Also, I'm assuming monitoring applications aren't aware of the turn off.
> 
> What happens to case a) ? Apps will need to get some error message as a reply. Some assembly overhead
> is saved (since such an error reply can be sent right away as soon as a query command is seen) but no
> substantial saving on the delivery part.
> 
> For case b), we can save on assembly and delivery by exiting early from taskstats_exit_send(). But won't
> we need to send some message (if not periodically, atleast once) to listening apps that they shouldn't
> expect any exit data ? Semantics of suddenly not seeing any exit data could be misinterpreted ?
> 
> Its easy enough to implement...just concerned about the semantics of doing so (as far as userspace
> apps are concerned) and utility in general settings. Utility in case where only CSA is running (delay
> accounting and other users turned off) is clear.
> 
> Thoughts ?

I would have thought that it was obvious that turning on/off the 
collection of statistics concerning the ACCRUED time that tasks spend in 
various states (which is what we're talking about) would cause the data 
to be horribly corrupted.

I also suspect that the overhead of the tests associated with checking 
whether the mechanism is on or off is probably almost as big as the 
overhead of gathering the statistics anyway.  Assuming that the clock 
time is already available (true for most places in the CPU scheduler 
where you'd want to gather stats, for one) the cost of gathering the 
stats is a subtraction, an addition and (possibly) an assignment to a 
time stamp (I say possibly here as you may get the time stamp updated 
for free as well).

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce