Re: [PATCH 0/6] support "dataplane" mode for nohz_full

Chris Metcalf
2015-05-26T19:52:11Z

Thanks for the clarification, and sorry for the slow reply; I had a busy
week of meetings last week, and then the long weekend in the U.S.

On 05/15/2015 02:44 PM, Mike Galbraith wrote:
> Just because the nohz_full feature itself is currently static is no
> reason to put users thereof in a straight jacket by mandating that any
> set they define irrevocably disappears from the generic resource pool .
> Those CPUS are useful until the moment someone cripples them, which
> making nohz_full imply isolcpus does if isolcpus then also becomes
> immutable, which Rik's patch does. Making nohz_full imply isolcpus
> sounds perfectly fine until someone comes along and makes isolcpus
> immutable (Rik's patch), at which point the user loses a choice due to
> two people making it imply things that _alone_ sound perfectly fine.
>
> See what I'm saying now?

That does make sense; my argument was that 99% of the time when
someone specifies nohz_full they also need isolcpus. You're right
that someone playing with nohz_full would be unpleasantly surprised.
And of course having more flexibility always feels like a plus.
On balance I suspect it's still better to make command line arguments
handle the common cases most succinctly.

Hopefully we'll get a to a point where all of this is dynamic and how
we play with the boot arguments no longer matters. If not, perhaps
we revisit this and make a cpu_isolation=1-15 type command line
argument that enables isolcpus and nohz_full both.

>>> Thomas has nuked the hrtimer softirq.
>> Yes, this I didn't know. So I will drop my "no ksoftirqd" patch and
>> we will see if ksoftirqs emerge as an issue for my "cpu isolation"
>> stuff in the future; it may be that that was the only issue.
>>
>>> Inlining softirqs may save a context switch, but adds cycles that we may
>>> consume at higher frequency than the thing we're avoiding.
>> Yes but consuming cycles is not nearly as much of a concern
>> as avoiding interrupts or scheduling, certainly for the case of
>> userspace drivers that I described above.
> If you're raising softirqs in an SMP kernel, you're also doing something
> that puts you at very serious risk of meeting the jitter monster, locks,
> and worse, sleeping locks, no?

The softirqs were being raised by third parties for hrtimer, not by
the application code itself, if I remember correctly. In any case
this appears not to be an issue for nohz_full any more now.

--
Chris Metcalf, EZChip Semiconductor