Re: SMTC support status in latest git head.

["Kevin D. Kissell" <kevink@paralogos.com>]

On 01/04/11 05:02, Anoop P A wrote:
> On Tue, 2011-01-04 at 00:17 -0800, Kevin D. Kissell wrote:
>> Those interrupt counters show that IPIs are being taken everywhere,
>> though very few by CPUs 5 and 6.  If I understand the configuration
>> correctly, CPU 4 is a TC in VPE 1, and it's getting a reasonable IPI
> Yes CPU4 is in second VPE
>
>> rate, *if* we're looking at a tickless kernel under low load.  But there
> No it was not the tickless kernel.I had selected 250 MHz timer. can't we
> expect IPI / timer interrupt for all the threads in this case ?.

In that case, you should expect a distribution of timer interrupts that 
favors the low-numbered TCs within the VPE, as you do in VPE0, and a 
distribution of IPIs that is sort-of the inverse, as you do in VPE0.  
But the low counts on VPE1 are indeed suspicious, as you note.

>> may be a clue there to part of your problem.  I have no idea why the
>> behavior would have changed from 2.6.36 to 2.6.37, but it looks as if
>> you're getting your clock interrupts through the MSP CIC interrupt
>> controller on VPE 0.  There's nothing symmetric for VPE1. The Malta
>> example code is perhaps deceptively simple, in that both VPEs have their
>> count/compare indication wired directly to the 2 clock interrupt inputs,
>> so that having both of them running with only a single set of irq state
>> just works.  I don't know whether the MSP CIC timer interrupt is a
> In my case it is separate irq. MSP_INT_VPE1_TIMER (34) and
> MSP_INT_VPE0_TIMER (25) are wired to CIC . CIC interrupt has been
> connected to cpu irq 6.
>
> I can reproduce cpu stall in VSMP mode If I don't setup VPE1 timer
> interrupt . Don't we have support for separate irq in SMTC
> implementation ?..

There are hooks for platform-specific SMTC support, which is implemented 
for the Malta in arch/mips/mti-malta/malta-smtc.c.  See 
msmtc_init_secondary(), for example, where the clock/compare, profile, 
and IPI interrupts are armed for VPE 1, while I/O peripheral interrupts 
are inhibited.

>> gating of the VPE0 count/compare output, or whether it's it's own
>> interval timer, but I suspect that you may need to do some further
>> low-level initialization in the platform-specific code to set up an
>> interrupt on the VPE1 side.  I don't think the snippet you've got below
>> would work as written.
> The routine which I copied works fine for VSMP mode .
>
> / # cat /proc/interrupts
>             CPU0       CPU1
>    0:        187        254            MIPS  IPI_resched
>    1:         77        174            MIPS  IPI_call
>    6:          0          0            MIPS  MSP CIC cascade
>    8:          0          0         MSP_CIC  Softreset button
>    9:          0          0         MSP_CIC  Standby switch
>   21:          0          0         MSP_CIC  MSP PER cascade
>   25:      37077          0         MSP_CIC  timer
>   27:        188          0         MSP_CIC  serial
>   34:          0      36986         MSP_CIC  timer
>
> Do I want to change anything specific for SMTC ? .

If it works (which I doubt), then we can critique stylistic points like 
using

		if ((1==get_current_vpe())

Instead of the more readable and general

		if (get_current_vpe()>  0)


But I think you're generally looking in the wrong place.  Look at the 
Malta code and see what's done where.  The initial SMTC code had a lot 
of Malta assumptions in the main line that I pushed out to platform code 
in later patches.  I can see how things could be made even more modular, 
but for the moment I think it's just that there's some stuff that ought 
to be done in a "msp_smtc.c" file that doesn't exist in 2.6.37.

             Regards,

             Kevin K.
>
>
>> If it's purely an issue with clock distribution on VPE1, then a boot
>> with maxvpes=1 maxtcs=4 should be stable.
> Yes the kernel seems to be stable if I boot with maxvpes=1 maxtcs=4 .
>
>> /K.
>>
>> On 1/3/2011 11:20 AM, Anoop P A wrote:
>>> Hi Kevin,
>>>
>>> On Mon, 2011-01-03 at 08:14 -0800, Kevin D. Kissell wrote:
>>>> The very first SMTC implementations didn't support full kernel-mode
>>>> preemption, which anyway wasn't a priority, given the hardware event
>>>> response support in MIPS MT.  I believe it was later made compatible,
>>>> but it was never extensively exercised.  Since SMTC has fingers in some
>>>> pretty low-level atomicity mechanisms, if a new, parallel set was
>>>> implemented for RCU, I can easily imagine that nobody has yet
>>>> implemented SMTC-ified variants of that set.
>>>>
>>>> Your last statement isn't very clear, though.  Are you saying that if
>>>> you configure for no forced preemption and with TREE_CPU, the 2.6.37
>>>> kernel boots all the way up, or that it simply hangs later?   What's the
>>>> last rev kernel that actually boots all the way up?
>>> I have debugged this a bit more. It seems that kernel getting stalled
>>> while executing on TC's of second VPE .
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=2504 jiffies)
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=10036 jiffies)
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=17568 jiffies)
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=25100 jiffies)
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=32632 jiffies)
>>> INFO: rcu_sched_state detected stalls on CPUs/tasks: { 4 5 6} (detected
>>> by 1, t=40164 jiffies)
>>>
>>> With CONFIG_TREE_CPU we were not hitting this scenario very often.
>>> However with CONFIG_PREEMPT_TREE_CPU stall happens most of the time.
>>>
>>> I presume some issue in my timer setup . I am not seeing timer interrupt
>>> (or IPI interrupt) getting  incremented for VPE1 tcs on a completely
>>> booted 2.6.32-stable kernel.
>>>
>>> / # cat /proc/interrupts
>>>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
>>> CPU6
>>>    1:        148      15023      15140      15093       3779          8
>>> 2            MIPS  SMTC_IPI
>>>    6:          0          0          0          0          0          0
>>> 0            MIPS  MSP CIC cascade
>>>    8:          0          0          0          0          0          0
>>> 0         MSP_CIC  Softreset button
>>>    9:          0          0          0          0          0          0
>>> 0         MSP_CIC  Standby switch
>>>   21:          0          0          0          0          0          0
>>> 0         MSP_CIC  MSP PER cascade
>>>   25:      15113        341          4          7          0          0
>>> 0         MSP_CIC  timer
>>>   27:        260          9          0          1          0          0
>>> 0         MSP_CIC  serial
>>>   34:          0          0          0          0          0          0
>>> 0         MSP_CIC  timer
>>>
>>> Can't we use separate timer interrupts for VPE1 and VPE0 in SMTC ?.
>>>
>>> I have tried setting up VPE1 timer from get_co_compare_int as follows
>>>
>>> unsigned int __cpuinit get_c0_compare_int(void)
>>> {
>>> 	if ((1==get_current_vpe())&&  !vpe1_timr_installed){
>>> 	
>>> 	memcpy(&timer_vpe1,&c0_compare_irqaction,sizeof(timer_vpe1));
>>> 	
>>> 	setup_irq(MSP_INT_VPE1_TIMER,&timer_vpe1);
>>>                    vpe1_timr_installed++;
>>>            }
>>>            return (get_current_vpe() ? MSP_INT_VPE1_TIMER :
>>> MSP_INT_VPE0_TIMER);
>>> }
>>>
>>> Thanks
>>> Anoop