Performance differences with IRQ_ALL_CPUS

Performance differences with IRQ_ALL_CPUS

[Subodh Nijsure]

Hi,


I have board with MPC8572E (dual core PPC). I have one board running kernel
with IRQ_ALL_CPUS set to Y and another with that option turned off.  Kernel
version #2.6.26


With IRQ_ALL_CPUS turned off
( Here interrupts all go to CPU0 )

hdparm -tT /dev/hda
/dev/hda:
 Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
 Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec

cat /proc/interrupts
           CPU0       CPU1
 18:    1394100          0   OpenPIC   Edge      ide0

With IRQ_ALL_CPUS turned on I see
(Here interrupts go to CPU0 and CPU1 )

hdparm -tT /dev/hda

/dev/hda:
 Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
 Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec

cat /proc/interrupts
           CPU0       CPU1
 18:      44951      54765   OpenPIC   Edge      ide0

I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be able to
get higher througput but I see things otherway around. 

Would someone care to enlighten me as to why when I set IRQ_ALL_CPUS disk
I/O performance goes down so much?  Under what circumstances should one then
turn on IRQ_ALL_CPUS option on PPC platform? 

Regards,
/Subodh

Re: Performance differences with IRQ_ALL_CPUS

[Kumar Gala]

On Apr 21, 2009, at 10:56 AM, Subodh Nijsure wrote:

> Hi,
>
>
> I have board with MPC8572E (dual core PPC). I have one board running  
> kernel
> with IRQ_ALL_CPUS set to Y and another with that option turned off.   
> Kernel
> version #2.6.26
>
>
> With IRQ_ALL_CPUS turned off
> ( Here interrupts all go to CPU0 )
>
> hdparm -tT /dev/hda
> /dev/hda:
> Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
> Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec
>
> cat /proc/interrupts
>           CPU0       CPU1
> 18:    1394100          0   OpenPIC   Edge      ide0
>
> With IRQ_ALL_CPUS turned on I see
> (Here interrupts go to CPU0 and CPU1 )
>
> hdparm -tT /dev/hda
>
> /dev/hda:
> Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
> Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec
>
> cat /proc/interrupts
>           CPU0       CPU1
> 18:      44951      54765   OpenPIC   Edge      ide0
>
> I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be  
> able to
> get higher througput but I see things otherway around.
>
> Would someone care to enlighten me as to why when I set IRQ_ALL_CPUS  
> disk
> I/O performance goes down so much?  Under what circumstances should  
> one then
> turn on IRQ_ALL_CPUS option on PPC platform?
>
> Regards,
> /Subodh

This is probably because you are getting ill defined behavior on  
8572.  The 8572 (and any FSL MPIC) doesn't actually support having the  
PIC distribute interrupts to the different CPUs.  Its technically  
"illegal" to set more than one destination processor.

If you want to try and distribute interrupts than I suggest looking at  
irqbalance.

I'll look at a patch to deal with IRQ_ALL_CPUS to not setup the insane  
HW scenario.

- k

RE: Performance differences with IRQ_ALL_CPUS

[Subodh Nijsure]

 

> From: Kumar Gala 
> Sent: Tuesday, April 21, 2009 1:02 PM

> 
> 
> On Apr 21, 2009, at 10:56 AM, Subodh Nijsure wrote:
> 
> > Hi,
> >
> >
> > I have board with MPC8572E (dual core PPC). I have one 
> board running 
> > kernel
> > with IRQ_ALL_CPUS set to Y and another with that option 
> turned off.   
> > Kernel
> > version #2.6.26
> >
> >
> > With IRQ_ALL_CPUS turned off
> > ( Here interrupts all go to CPU0 )
> >
> > hdparm -tT /dev/hda
> > /dev/hda:
> > Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
> > Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec
> >
> > cat /proc/interrupts
> >           CPU0       CPU1
> > 18:    1394100          0   OpenPIC   Edge      ide0
> >
> > With IRQ_ALL_CPUS turned on I see
> > (Here interrupts go to CPU0 and CPU1 )
> >
> > hdparm -tT /dev/hda
> >
> > /dev/hda:
> > Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
> > Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec
> >
> > cat /proc/interrupts
> >           CPU0       CPU1
> > 18:      44951      54765   OpenPIC   Edge      ide0
> >
> > I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be 
> > able to get higher througput but I see things otherway around.
> >
> > Would someone care to enlighten me as to why when I set 
> IRQ_ALL_CPUS 
> > disk I/O performance goes down so much?  Under what circumstances 
> > should one then turn on IRQ_ALL_CPUS option on PPC platform?
> >
> > Regards,
> > /Subodh
> 
> This is probably because you are getting ill defined behavior 
> on 8572.  The 8572 (and any FSL MPIC) doesn't actually 
> support having the PIC distribute interrupts to the different 
> CPUs.  Its technically "illegal" to set more than one 
> destination processor.
> 
> If you want to try and distribute interrupts than I suggest 
> looking at irqbalance.
> 

Kumar, thanks for the info.
Okay I will research irqbalance (from http://irqbalance.org/?) and see how
that goes.

If I want to lock say ethernet interrupts to (core0) CPU0 and Ide interrupts
to (core1) CPU1 is there a way to fix that using device-tree specification?
I will look through the MPC8572 programmer's manual and see if there is such
a thing in processor itself.


/Subodh

Re: Performance differences with IRQ_ALL_CPUS

[Kumar Gala]

On Apr 21, 2009, at 5:01 PM, Subodh Nijsure wrote:

>
>
>> From: Kumar Gala
>> Sent: Tuesday, April 21, 2009 1:02 PM
>
>>
>>
>> On Apr 21, 2009, at 10:56 AM, Subodh Nijsure wrote:
>>
>>> Hi,
>>>
>>>
>>> I have board with MPC8572E (dual core PPC). I have one
>> board running
>>> kernel
>>> with IRQ_ALL_CPUS set to Y and another with that option
>> turned off.
>>> Kernel
>>> version #2.6.26
>>>
>>>
>>> With IRQ_ALL_CPUS turned off
>>> ( Here interrupts all go to CPU0 )
>>>
>>> hdparm -tT /dev/hda
>>> /dev/hda:
>>> Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
>>> Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec
>>>
>>> cat /proc/interrupts
>>>          CPU0       CPU1
>>> 18:    1394100          0   OpenPIC   Edge      ide0
>>>
>>> With IRQ_ALL_CPUS turned on I see
>>> (Here interrupts go to CPU0 and CPU1 )
>>>
>>> hdparm -tT /dev/hda
>>>
>>> /dev/hda:
>>> Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
>>> Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec
>>>
>>> cat /proc/interrupts
>>>          CPU0       CPU1
>>> 18:      44951      54765   OpenPIC   Edge      ide0
>>>
>>> I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be
>>> able to get higher througput but I see things otherway around.
>>>
>>> Would someone care to enlighten me as to why when I set
>> IRQ_ALL_CPUS
>>> disk I/O performance goes down so much?  Under what circumstances
>>> should one then turn on IRQ_ALL_CPUS option on PPC platform?
>>>
>>> Regards,
>>> /Subodh
>>
>> This is probably because you are getting ill defined behavior
>> on 8572.  The 8572 (and any FSL MPIC) doesn't actually
>> support having the PIC distribute interrupts to the different
>> CPUs.  Its technically "illegal" to set more than one
>> destination processor.
>>
>> If you want to try and distribute interrupts than I suggest
>> looking at irqbalance.
>>
>
> Kumar, thanks for the info.
> Okay I will research irqbalance (from http://irqbalance.org/?) and  
> see how
> that goes.
>
> If I want to lock say ethernet interrupts to (core0) CPU0 and Ide  
> interrupts
> to (core1) CPU1 is there a way to fix that using device-tree  
> specification?

No. We've actively avoided trying to convene such affinity in the  
device tree.

However you can do:

cat 1 > /proc/irq/ENET_IRQ_NUM/affinity
cat 2 > /proc/irq/IDE_IRQ/affinity

at boot time.

This will get you affinity binding to a core.

The affinity file is bitmask related to core # (with lsb being core0  
and msb being core31)

- k