Re: [PATCH] i386 No-Idle-Hz aka Dynamic-Ticks 3

[George Anzinger <george@mvista.com>]

Tony Lindgren wrote:
> * Srivatsa Vaddagiri <vatsa@in.ibm.com> [050805 05:37]:
> 
>>On Wed, Aug 03, 2005 at 06:05:28AM +0000, Con Kolivas wrote:
>>
>>>This is the dynamic ticks patch for i386 as written by Tony Lindgen 
>>><tony@atomide.com> and Tuukka Tikkanen <tuukka.tikkanen@elektrobit.com>. 
>>>Patch for 2.6.13-rc5
>>>
>>>There were a couple of things that I wanted to change so here is an updated 
>>>version. This code should have stabilised enough for general testing now.
>>
>>Con,
>>	I have been looking at some of the requirement of tickless idle CPUs in
>>core kernel areas like scheduler and RCU. Basically, both power management and 
>>virtualization benefit if idle CPUs can cut off useless timer ticks. Especially 
>>from a virtualization standpoint, I think it makes sense that we enable this 
>>feature on a per-CPU basis i.e let individual CPUs cut off their ticks as and 
>>when they become idle. The benefit of this is more visible in platforms that 
>>host lot of (SMP) VMs on the same machine. Most of the time, these VMs may be 
>>partially idle (some CPUs in it are idle, some not) and it is good that we 
>>quiesce the timer ticks on the partial set of idle CPUs. Both S390 and Xen ports
>>of Linux kernel have this ability today (S390 has it in mainline already and 
>>Xen has it out of tree).
> 
> 
> Good point, and it would be nice to have it resolved for systems that support
> idling individual CPUs. The current setup was done because when I was tinkering
> with the amd76x_pm patch a while a back, I noticed that idling the cpu
> disconnects all cpus from the bus. (As far as I remember)
> 
> So this may need to be configured depending on the system.
> 
> 
>>From this viewpoint, I think the current implementation of dynamic tick
>>falls short of this requirement. It cuts of the timer ticks only when 
>>all CPUs go idle.
>>
>>Apart from this observation, I have some others about the current dynamic tick
>>patch:
>>
>>- All CPUs seem to cut off the same number of ticks (dyn_tick->skip). Isn't
>>  this wrong, considering that the timer list is per-CPU? This will cause
>>  some timers to be serviced much later than usual.
> 
> 
> Yes if it's done on per-CPU basis. In the current setup the first interrupt
> will kick the system off the dyn-tick state and the timers get checked again.
> 
> 
>>- The fact that dyn_tick_state is global and accessed from all CPUs
>>  is probably a scalability concern, especially if we allow the ticks
>>  to be cut off on per-CPU basis.
> 
> 
>>From idling devices point of view, we still need some global variable I
> believe. How else would you be able to tell all devices that the whole
> system does not have any timers for next 2 seconds?
> 
> 
>>- Again, when we allow this on a per-CPU basis, subsystems like
>>  RCU need to know the partial set of idle CPUs. RCU already does
>>  that thr' nohz_cpu_mask (which will need to replace dyn_cpu_map).
> 
> 
> Sounds like that could work for dyn-tick too.
> 
> 
>>- Looking at dyn_tick_timer_interrupt, would it be nice if we avoid calling 
>>  do_timer_interrupt so many times and instead update jiffies to
>>  (skipped_ticks - 1) and then call do_timer_interrupt once? I think
>>  VST does it that way.
> 
> 
> In the long run we would do the calculations in usecs and just emulate
> jiffies from the hw timer. But yes, optimizing updating the time would be
> great.
> 
> 
>>- dyn_tick->max_skip = 0xffffff / apic_timer_val;
>>	From my reading of Intel docs, APIC_TMICT is 32-bit. So why does the
>>  above calculation take only 24-bits into account? What am I missing here?
> 
> 
> Hmm, could be a bug here, needs to be checked. Maybe 32-bit APIC timer is
> optional support, or maybe I accidentally pulled the optional 24-bit support
> from the ACPI PM timer.
> 
> But in any case on P4 systems the APIC timer is not the bottleneck as
> stopping or reprogramming PIT also kills APIC. (This does not happen on P3
> systems). So the bottleneck most likely is the length of PIT.
> 
> 
>>I can take a shot at addressing these concerns in dynamic_tick patch, but it 
>>seems to me that VST has already addressed all these to a big extent. Had you 
>>considered VST before? The biggest bottleneck I see in VST going mainline is 
>>its dependency on HRT patch but IMO it should be possible to write a small patch
>>to support VST w/o HRT. 
>>
>>George, what do you think?
> 
> 
> HRT + VST depend on APIC only, and does not use next_timer_interrupt().

I convinced my self that the next_timer... code in timer.c misses timers 
(i.e. gives the wrong answer).  I did this (after wondering due to 
performance) by scanning the whole timer list after I had the 
next_timer... answer and finding a better answer, not always, but some 
times.  That code does not address the cascade list correctly.


-- 
George Anzinger   george@mvista.com
HRT (High-res-timers):  http://sourceforge.net/projects/high-res-timers/