Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Hi,

I'm using a Technisat Skystar HD2 (DVB-S2) with a CI Module under Ubuntu 
11.04.
As some people already noticed, the mantis_ca_init() is never called to 
initialize the CAM.
Since s2-liplianin used almost the same code, I basically just put the 
mantis_ca_init back in,
which is working quite good. But I hope somebody can help me to remove a 
bug rendering the driver not SMP safe,
since I believe my work around for this makes the driver less reliable.

First of all the description of the bug:
I'm using a dual core cpu and noticed that I don't get all the interrupt 
i should get when writing to/ reading from the card using a function 
which uses "mantis_hif_sbuf_opdone_wait" in "mantis_hif.c".
This leads to the 500 ms timeout. Interesting enough, when reading the 
data despite the timeout, the data is valid and available. Using 
max_cpus=1 parameter when starting ubuntu 11.04 solves the problem; all 
interrupts are received and no timeout occurs.
In addition to this, i think the return value of "msecs_to_jiffies" 
changed with some kernel update an thus "mantis_hif_sbuf_opdone_wait" 
never returns an error.
How hope someone can help figuraing out, why the card send less 
interrupt on SMP enabled machines. I know the core which handles the IRQ 
can change, but even all the IRQs from all core are less than when 
disabling SMP.

Now the description how I added the CI support again:

File mantis_hif.c (workaround for the SMP bug):
- Change the call from msecs_to_jiffies(500) to msecs_to_jiffies(2) in 
function "mantis_hif_sbuf_opdone_wait" (we just get the data after 2 ms, 
regardless if we got the data ready IRQ or not).

File mantis_pci.c:
- Move the function set_direction from mantis_core.c to mantis_pci.c (I 
tried to just add the forward declaration to mantis_core.h, but I 
couldn't get it to work...)
- Add its function declaration to mantis_pci.h (extern void 
mantis_set_direction(struct mantis_pci *mantis, int direction);).
- Add "mantis_set_direction(mantis, 0);" after "mantis->revision = 
pdev->revision;" in function "mantis_pci_init".
- Add "mmwrite(0x00, MANTIS_INT_MASK);" before "err = 
request_irq(pdev->irq,"... in function "mantis_pci_init".

File mantis_ca.c:
- Add the include #include "mantis_pci.h"
- Comment in "mantis_set_direction(mantis, 1);" in function 
"mantis_ts_control" in file mantis_ca.c

File manits_dvb.c:
- Add the function call "mantis_ca_init(mantis);" right before the 
return 0 in function "mantis_dvb_init".
- Add th function call "mantis_ca_exit(mantis);" right before 
"tasklet_kill(&mantis->tasklet);" in function "mantis_dvb_exit".

Regards,
Manuel

Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Hi,

I'm using a Technisat Skystar HD2 (DVB-S2) with a CI Module under Ubuntu 
11.04.
As some people already noticed, the mantis_ca_init() is never called to 
initialize the CAM.
Since s2-liplianin used almost the same code, I basically just put the 
mantis_ca_init back in,
which is working quite good. But I hope somebody can help me to remove a 
bug rendering the driver not SMP safe,
since I believe my work around for this makes the driver less reliable.

First of all the description of the bug:
I'm using a dual core cpu and noticed that I don't get all the interrupt 
i should get when writing to/ reading from the card using a function 
which uses "mantis_hif_sbuf_opdone_wait" in "mantis_hif.c".
This leads to the 500 ms timeout. Interesting enough, when reading the 
data despite the timeout, the data is valid and available. Using 
max_cpus=1 parameter when starting ubuntu 11.04 solves the problem; all 
interrupts are received and no timeout occurs.
In addition to this, i think the return value of "msecs_to_jiffies" 
changed with some kernel update an thus "mantis_hif_sbuf_opdone_wait" 
never returns an error.
How hope someone can help figuraing out, why the card send less 
interrupt on SMP enabled machines. I know the core which handles the IRQ 
can change, but even all the IRQs from all core are less than when 
disabling SMP.

Now the description how I added the CI support again:

File mantis_hif.c (workaround for the SMP bug):
- Change the call from msecs_to_jiffies(500) to msecs_to_jiffies(2) in 
function "mantis_hif_sbuf_opdone_wait" (we just get the data after 2 ms, 
regardless if we got the data ready IRQ or not).

File mantis_pci.c:
- Move the function set_direction from mantis_core.c to mantis_pci.c (I 
tried to just add the forward declaration to mantis_core.h, but I 
couldn't get it to work...)
- Add its function declaration to mantis_pci.h (extern void 
mantis_set_direction(struct mantis_pci *mantis, int direction);).
- Add "mantis_set_direction(mantis, 0);" after "mantis->revision = 
pdev->revision;" in function "mantis_pci_init".
- Add "mmwrite(0x00, MANTIS_INT_MASK);" before "err = 
request_irq(pdev->irq,"... in function "mantis_pci_init".

File mantis_ca.c:
- Add the include #include "mantis_pci.h"
- Comment in "mantis_set_direction(mantis, 1);" in function 
"mantis_ts_control" in file mantis_ca.c

File manits_dvb.c:
- Add the function call "mantis_ca_init(mantis);" right before the 
return 0 in function "mantis_dvb_init".
- Add th function call "mantis_ca_exit(mantis);" right before 
"tasklet_kill(&mantis->tasklet);" in function "mantis_dvb_exit".

Regards,
Manuel

Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Hi,

I'm using a Technisat Skystar HD2 (DVB-S2) with a CI Module under Ubuntu 
11.04.
As some people already noticed, the mantis_ca_init() is never called to 
initialize the CAM.
Since s2-liplianin used almost the same code, I basically just put the 
mantis_ca_init back in,
which is working quite good. But I hope somebody can help me to remove a 
bug rendering the driver not SMP safe,
since I believe my work around for this makes the driver less reliable.

First of all the description of the bug:
I'm using a dual core cpu and noticed that I don't get all the interrupt 
i should get when writing to/ reading from the card using a function 
which uses "mantis_hif_sbuf_opdone_wait" in "mantis_hif.c".
This leads to the 500 ms timeout. Interesting enough, when reading the 
data despite the timeout, the data is valid and available. Using 
max_cpus=1 parameter when starting ubuntu 11.04 solves the problem; all 
interrupts are received and no timeout occurs.
In addition to this, i think the return value of "msecs_to_jiffies" 
changed with some kernel update an thus "mantis_hif_sbuf_opdone_wait" 
never returns an error.
How hope someone can help figuraing out, why the card send less 
interrupt on SMP enabled machines. I know the core which handles the IRQ 
can change, but even all the IRQs from all core are less than when 
disabling SMP.

Now the description how I added the CI support again:

File mantis_hif.c (workaround for the SMP bug):
- Change the call from msecs_to_jiffies(500) to msecs_to_jiffies(2) in 
function "mantis_hif_sbuf_opdone_wait" (we just get the data after 2 ms, 
regardless if we got the data ready IRQ or not).

File mantis_pci.c:
- Move the function set_direction from mantis_core.c to mantis_pci.c (I 
tried to just add the forward declaration to mantis_core.h, but I 
couldn't get it to work...)
- Add its function declaration to mantis_pci.h (extern void 
mantis_set_direction(struct mantis_pci *mantis, int direction);).
- Add "mantis_set_direction(mantis, 0);" after "mantis->revision = 
pdev->revision;" in function "mantis_pci_init".
- Add "mmwrite(0x00, MANTIS_INT_MASK);" before "err = 
request_irq(pdev->irq,"... in function "mantis_pci_init".

File mantis_ca.c:
- Add the include #include "mantis_pci.h"
- Comment in "mantis_set_direction(mantis, 1);" in function 
"mantis_ts_control" in file mantis_ca.c

File manits_dvb.c:
- Add the function call "mantis_ca_init(mantis);" right before the 
return 0 in function "mantis_dvb_init".
- Add th function call "mantis_ca_exit(mantis);" right before 
"tasklet_kill(&mantis->tasklet);" in function "mantis_dvb_exit".

Regards,
Manuel

Re: Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Hi,

has anyone an idea how the SMP problems could be fixed?
I did some further investigation. When comparing the number of 
interrupts with all cores enabled and the interrupts with only one core 
enabled it seems like only the IRQ0 changed, the other IRQs and the 
total number stays quite the same:

4 Cores:
All IRQ/sec: 493
Masked IRQ/sec: 400
Unknown IRQ/sec: 0
DMA/sec: 400
IRQ-0/sec: 143
IRQ-1/sec: 0
OCERR/sec: 0
PABRT/sec: 0
RIPRR/sec: 0
PPERR/sec: 0
FTRGT/sec: 0
RISCI/sec: 258
RACK/sec: 0

1 Core:
All IRQ/sec: 518
Masked IRQ/sec: 504
Unknown IRQ/sec: 0
DMA/sec: 504
IRQ-0/sec: 246
IRQ-1/sec: 0
OCERR/sec: 0
PABRT/sec: 0
RIPRR/sec: 0
PPERR/sec: 0
FTRGT/sec: 0
RISCI/sec: 258
RACK/sec: 0

So, where might be the problem?
I don't think the IRQ gets lost on the way from the device to the 
driver, because only IRQ0 is affected.
I don't think the device or the driver is to slow because it works fine 
with only one Core and it also works with
SMP when ignoring the timeout and just read the data at the time the IRQ 
should have fired.
Maybe the (cam) locking is faulty (looks fine to me...).
Maybe the IRQ handler gets executed parallel on two cores which leads to 
the problem. But then I think this
should be fixed when setting IRQ affinity to only core, but it didn't 
help with the problem.

I hope somebody can help, because I think we are very close to a fully 
functional CAM here.
I ran out of things to test to get closer to the solution :(
Btw: Is there any documentation available for the mantis PCI bridge?

Manuel

Re: Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Marko Ristola]

On 12/10/2011 01:57 AM, Ninja wrote:
> Hi,
>
> has anyone an idea how the SMP problems could be fixed?

You could turn on Mantis Kernel module's debug messages.
It could tell you the emitted interrupts.

One risky thing with the Interrupt handler code is that
MANTIS_GPIF_STATUS is cleared, even though IRQ0 isn't active yet.
This could lead to a rare starvation of the wait queue you described.
I supplied a patch below. Does it help?

> I did some further investigation. When comparing the number of interrupts with all cores enabled and the interrupts with only one core enabled it seems like only the IRQ0 changed, the other IRQs and the total number stays quite the same:
>
> 4 Cores:
> All IRQ/sec: 493
> Masked IRQ/sec: 400
> Unknown IRQ/sec: 0
> DMA/sec: 400
> IRQ-0/sec: 143
> IRQ-1/sec: 0
> OCERR/sec: 0
> PABRT/sec: 0
> RIPRR/sec: 0
> PPERR/sec: 0
> FTRGT/sec: 0
> RISCI/sec: 258
> RACK/sec: 0
>
> 1 Core:
> All IRQ/sec: 518
> Masked IRQ/sec: 504
> Unknown IRQ/sec: 0
> DMA/sec: 504
> IRQ-0/sec: 246
> IRQ-1/sec: 0
> OCERR/sec: 0
> PABRT/sec: 0
> RIPRR/sec: 0
> PPERR/sec: 0
> FTRGT/sec: 0
> RISCI/sec: 258
> RACK/sec: 0
>
> So, where might be the problem?
Turning on Mantis debug messages, might tell the difference between these interrupts.

....
> I hope somebody can help, because I think we are very close to a fully functional CAM here.
> I ran out of things to test to get closer to the solution :(
> Btw: Is there any documentation available for the mantis PCI bridge?
Not that I know.

>
> Manuel
>
>
>
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


Regards,
Marko Ristola

----------------------- PATCH ------------------------------
Mantis/Hopper: Check and clear GPIF status bits only when IRQ0 bit is active.

Signed-off-by: Marko Ristola <Marko.Ristola@kolumbus.fi>

diff --git a/drivers/media/dvb/mantis/hopper_cards.c b/drivers/media/dvb/mantis/hopper_cards.c
index 71622f6..c2084e9 100644
--- a/drivers/media/dvb/mantis/hopper_cards.c
+++ b/drivers/media/dvb/mantis/hopper_cards.c
@@ -84,15 +84,6 @@ static irqreturn_t hopper_irq_handler(int irq, void *dev_id)
  	if (!(stat & mask))
  		return IRQ_NONE;
  
-	rst_mask  = MANTIS_GPIF_WRACK  |
-		    MANTIS_GPIF_OTHERR |
-		    MANTIS_SBUF_WSTO   |
-		    MANTIS_GPIF_EXTIRQ;
-
-	rst_stat  = mmread(MANTIS_GPIF_STATUS);
-	rst_stat &= rst_mask;
-	mmwrite(rst_stat, MANTIS_GPIF_STATUS);
-
  	mantis->mantis_int_stat = stat;
  	mantis->mantis_int_mask = mask;
  	dprintk(MANTIS_DEBUG, 0, "\n-- Stat=<%02x> Mask=<%02x> --", stat, mask);
@@ -101,6 +92,16 @@ static irqreturn_t hopper_irq_handler(int irq, void *dev_id)
  	}
  	if (stat & MANTIS_INT_IRQ0) {
  		dprintk(MANTIS_DEBUG, 0, "<%s>", label[1]);
+
+		rst_mask  = MANTIS_GPIF_WRACK  |
+			    MANTIS_GPIF_OTHERR |
+			    MANTIS_SBUF_WSTO   |
+			    MANTIS_GPIF_EXTIRQ;
+
+		rst_stat  = mmread(MANTIS_GPIF_STATUS);
+		rst_stat &= rst_mask;
+		mmwrite(rst_stat, MANTIS_GPIF_STATUS);
+
  		mantis->gpif_status = rst_stat;
  		wake_up(&ca->hif_write_wq);
  		schedule_work(&ca->hif_evm_work);
diff --git a/drivers/media/dvb/mantis/mantis_cards.c b/drivers/media/dvb/mantis/mantis_cards.c
index c2bb90b..109a5fb 100644
--- a/drivers/media/dvb/mantis/mantis_cards.c
+++ b/drivers/media/dvb/mantis/mantis_cards.c
@@ -92,15 +92,6 @@ static irqreturn_t mantis_irq_handler(int irq, void *dev_id)
  	if (!(stat & mask))
  		return IRQ_NONE;
  
-	rst_mask  = MANTIS_GPIF_WRACK  |
-		    MANTIS_GPIF_OTHERR |
-		    MANTIS_SBUF_WSTO   |
-		    MANTIS_GPIF_EXTIRQ;
-
-	rst_stat  = mmread(MANTIS_GPIF_STATUS);
-	rst_stat &= rst_mask;
-	mmwrite(rst_stat, MANTIS_GPIF_STATUS);
-
  	mantis->mantis_int_stat = stat;
  	mantis->mantis_int_mask = mask;
  	dprintk(MANTIS_DEBUG, 0, "\n-- Stat=<%02x> Mask=<%02x> --", stat, mask);
@@ -109,6 +100,15 @@ static irqreturn_t mantis_irq_handler(int irq, void *dev_id)
  	}
  	if (stat & MANTIS_INT_IRQ0) {
  		dprintk(MANTIS_DEBUG, 0, "<%s>", label[1]);
+		rst_mask  = MANTIS_GPIF_WRACK  |
+			    MANTIS_GPIF_OTHERR |
+			    MANTIS_SBUF_WSTO   |
+			    MANTIS_GPIF_EXTIRQ;
+
+		rst_stat  = mmread(MANTIS_GPIF_STATUS);
+		rst_stat &= rst_mask;
+		mmwrite(rst_stat, MANTIS_GPIF_STATUS);
+
  		mantis->gpif_status = rst_stat;
  		wake_up(&ca->hif_write_wq);
  		schedule_work(&ca->hif_evm_work);

Re: Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Am 12.12.2011 12:08, schrieb Marko Ristola:
> On 12/10/2011 01:57 AM, Ninja wrote:
>> Hi,
>>
>> has anyone an idea how the SMP problems could be fixed?
>
> You could turn on Mantis Kernel module's debug messages.
> It could tell you the emitted interrupts.
>
> One risky thing with the Interrupt handler code is that
> MANTIS_GPIF_STATUS is cleared, even though IRQ0 isn't active yet.
> This could lead to a rare starvation of the wait queue you described.
> I supplied a patch below. Does it help?
>
>> I did some further investigation. When comparing the number of 
>> interrupts with all cores enabled and the interrupts with only one 
>> core enabled it seems like only the IRQ0 changed, the other IRQs and 
>> the total number stays quite the same:
>>
>> 4 Cores:
>> All IRQ/sec: 493
>> Masked IRQ/sec: 400
>> Unknown IRQ/sec: 0
>> DMA/sec: 400
>> IRQ-0/sec: 143
>> IRQ-1/sec: 0
>> OCERR/sec: 0
>> PABRT/sec: 0
>> RIPRR/sec: 0
>> PPERR/sec: 0
>> FTRGT/sec: 0
>> RISCI/sec: 258
>> RACK/sec: 0
>>
>> 1 Core:
>> All IRQ/sec: 518
>> Masked IRQ/sec: 504
>> Unknown IRQ/sec: 0
>> DMA/sec: 504
>> IRQ-0/sec: 246
>> IRQ-1/sec: 0
>> OCERR/sec: 0
>> PABRT/sec: 0
>> RIPRR/sec: 0
>> PPERR/sec: 0
>> FTRGT/sec: 0
>> RISCI/sec: 258
>> RACK/sec: 0
>>
>> So, where might be the problem?
> Turning on Mantis debug messages, might tell the difference between 
> these interrupts.
>
> ....
>> I hope somebody can help, because I think we are very close to a 
>> fully functional CAM here.
>> I ran out of things to test to get closer to the solution :(
>> Btw: Is there any documentation available for the mantis PCI bridge?
> Not that I know.
>
>>
>> Manuel
>>
>>
>>
>>
>>
>>
>>
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
> Regards,
> Marko Ristola
>

Hi Marko,

thanks for the patch. I did some quick testing today. The IRQ0 problem 
stays, but it seems like the small hangs (3-5 seconds every 20 minutes 
or something) are fixed :)

Manuel

Re: Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 (DVB-S2)

[Ninja]

Am 13.12.2011 22:30, schrieb Ninja:
> Am 12.12.2011 12:08, schrieb Marko Ristola:
>> On 12/10/2011 01:57 AM, Ninja wrote:
>>> Hi,
>>>
>>> has anyone an idea how the SMP problems could be fixed?
>>
>> You could turn on Mantis Kernel module's debug messages.
>> It could tell you the emitted interrupts.
>>
>> One risky thing with the Interrupt handler code is that
>> MANTIS_GPIF_STATUS is cleared, even though IRQ0 isn't active yet.
>> This could lead to a rare starvation of the wait queue you described.
>> I supplied a patch below. Does it help?
>>
>>> I did some further investigation. When comparing the number of 
>>> interrupts with all cores enabled and the interrupts with only one 
>>> core enabled it seems like only the IRQ0 changed, the other IRQs and 
>>> the total number stays quite the same:
>>>
>>> 4 Cores:
>>> All IRQ/sec: 493
>>> Masked IRQ/sec: 400
>>> Unknown IRQ/sec: 0
>>> DMA/sec: 400
>>> IRQ-0/sec: 143
>>> IRQ-1/sec: 0
>>> OCERR/sec: 0
>>> PABRT/sec: 0
>>> RIPRR/sec: 0
>>> PPERR/sec: 0
>>> FTRGT/sec: 0
>>> RISCI/sec: 258
>>> RACK/sec: 0
>>>
>>> 1 Core:
>>> All IRQ/sec: 518
>>> Masked IRQ/sec: 504
>>> Unknown IRQ/sec: 0
>>> DMA/sec: 504
>>> IRQ-0/sec: 246
>>> IRQ-1/sec: 0
>>> OCERR/sec: 0
>>> PABRT/sec: 0
>>> RIPRR/sec: 0
>>> PPERR/sec: 0
>>> FTRGT/sec: 0
>>> RISCI/sec: 258
>>> RACK/sec: 0
>>>
>>> So, where might be the problem?
>> Turning on Mantis debug messages, might tell the difference between 
>> these interrupts.
>>
>> ....
>>> I hope somebody can help, because I think we are very close to a 
>>> fully functional CAM here.
>>> I ran out of things to test to get closer to the solution :(
>>> Btw: Is there any documentation available for the mantis PCI bridge?
>> Not that I know.
>>
>>>
>>> Manuel
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-media" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>> Regards,
>> Marko Ristola
>>
>
> Hi Marko,
>
> thanks for the patch. I did some quick testing today. The IRQ0 problem 
> stays, but it seems like the small hangs (3-5 seconds every 20 minutes 
> or something) are fixed :)
>
> Manuel

Hi,

I did some further investigation of my problem. Almost all IRQ0s 
originate from calling the function "mantis_hif_read_iom" (at least when 
the CAM is up and running). Changing the udelay between the writes to 
about 100 gets almost rid of the lost IRQ0 problem, but somehow it 
increases the number of total interrupts and IRQ0 as well to about 
double to triple of the numbers with udelay(20).
This increase doesn't happen when reducing the number of cores as 
workaround.
And getting *almost* no timeouts doesn't help much, because every 
timeout causes a hang/freeze until the CAM is initialized again.
Changing the PCI latency to 0xff didn't help either.

btw: The DMA patches of Marko postet in the other thread "Multiple 
Mantis devices gives me glitches" doesn't help me further since I'm 
using the latest code which already includes the patch.

Manuel