Multiple Mantis devices gives me glitches

Multiple Mantis devices gives me glitches

[Vidar Tyldum]

Hello,

I have three  Cinergy C (DVB-C cards) like this:
05:04.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI
Bridge Controller [Ver 1.0] (rev 01)
        Subsystem: TERRATEC Electronic GmbH Device 1178
        Flags: bus master, medium devsel, latency 128, IRQ 20
        Memory at fdcfe000 (32-bit, prefetchable) [size=4K]
        Kernel driver in use: Mantis
        Kernel modules: mantis

Kernel: 2.6.38-13-generic-pae (Ubuntu Natty stock)
Motherboard: P43-ES3G
CPU: Intel(R) Core(TM)2 Quad CPU    Q8400

At some point i started having glitches (I would from time to time get an
'old' frame displayed and sometimes audio noise when this happened). I tried
pretty much every trick I could find:
 * CPU affinity
 * Dedicated IRQ for each card (only shared with USB, which has no units
attached)
 * Various process priorities (also for the kdvb-processes)
 * pci latency (from 0x20 to 0xff)

I have quite decent results when I only have 2 DVB cards present, and the
results became even better when running the irqbalancer-dæmon as well.
The glitches are not completely gone, but much more manageble now.

So the problem seems to be caused by too many interrupts for my system to
handle, however this is where I am in over my head.

I know 2.6.38 isn't the freshest brew, but I could not find any changes to
the driver since then that seemed relevant (which could just be my lack of
source-fu).

So, any ideas on how to improve the performance? I am suffering from some
hardware incompatibility or is the driver this resource hungry?



-- 
Vidar Tyldum
                              vidar@tyldum.com               PGP: 0x3110AA98

Re: Multiple Mantis devices gives me glitches

[Ninja]

> Hello,
>
> I have three  Cinergy C (DVB-C cards) like this:
> 05:04.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI
> Bridge Controller [Ver 1.0] (rev 01)
>          Subsystem: TERRATEC Electronic GmbH Device 1178
>          Flags: bus master, medium devsel, latency 128, IRQ 20
>          Memory at fdcfe000 (32-bit, prefetchable) [size=4K]
>          Kernel driver in use: Mantis
>          Kernel modules: mantis
>
> Kernel: 2.6.38-13-generic-pae (Ubuntu Natty stock)
> Motherboard: P43-ES3G
> CPU: Intel(R) Core(TM)2 Quad CPU    Q8400
>
> At some point i started having glitches (I would from time to time get an
> 'old' frame displayed and sometimes audio noise when this happened). I tried
> pretty much every trick I could find:
>   * CPU affinity
>   * Dedicated IRQ for each card (only shared with USB, which has no units
> attached)
>   * Various process priorities (also for the kdvb-processes)
>   * pci latency (from 0x20 to 0xff)
>
> I have quite decent results when I only have 2 DVB cards present, and the
> results became even better when running the irqbalancer-dæmon as well.
> The glitches are not completely gone, but much more manageble now.
>
> So the problem seems to be caused by too many interrupts for my system to
> handle, however this is where I am in over my head.
>
> I know 2.6.38 isn't the freshest brew, but I could not find any changes to
> the driver since then that seemed relevant (which could just be my lack of
> source-fu).
>
> So, any ideas on how to improve the performance? I am suffering from some
> hardware incompatibility or is the driver this resource hungry?

Hi,
I noticed some SMP problems with the mantis driver as well (see my post 
"Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2 
(DVB-S2)").
One workaround for me is to limit the CPU to one core (to be sure 
disable the hyperthreading cores as well). That can be done via BIOS 
*or* adding maxcpus=1 as kernel parameter *or* you can disable the cores 
one by one via "|echo 0 > /sys/devices/system/cpu/cpuX/online|" where X 
is the core to disable. Since you need to be root for this, I did "sudo 
su" first.
But of course our problems might be completely unrelated and limiting to 
one core won't change a thing ;)

Manuel

Re: Multiple Mantis devices gives me glitches

[Vidar Tyldum]

(sorry for breaking the threading, the reply came faster than my
subscription to the list :)

>From Ninja <Ninja15@xxxxxx>,
> Hi, I noticed some SMP problems with the mantis driver as well (see my
> post "Mantis CAM not SMP safe / Activating CAM on Technisat Skystar HD2
> (DVB-S2)"). One workaround for me is to limit the CPU to one core (to be
> sure disable the hyperthreading cores as well). That can be done via BIOS
> *or* adding maxcpus=1 as kernel parameter *or* you can disable the cores
> one by one via "|echo 0 > /sys/devices/system/cpu/cpuX/online|" where X
> is the core to disable. Since you need to be root for this, I did "sudo
> su" first. But of course our problems might be completely unrelated and
> limiting to one core won't change a thing ;)
> 
> Manuel

I doubt I'll revert back to the stock mantis module to test right now, but
your suggestion and observations might certainly be of interest. If/when
Ubuntu releases a new kernel which overwrites my current driver I will give
it a go.
However, instead of disabling SMP completely I hope adjusting IRQ affinity
could be sufficient:
  http://kernel.org/doc/Documentation/IRQ-affinity.txt

Couple this with increased PCI latency might work, however with three
devices connected I think the only solution is to reduce the number of
interrupts (though I am by no means any expert on this subject).

-- 
Vidar Tyldum
                              vidar@tyldum.com               PGP: 0x3110AA98

Re: Multiple Mantis devices gives me glitches

[Marko Ristola]

Hi

Here is a patch that went into Linus GIT this year.
It reduces the number of DMA transfer interrupts into one third.
Linus released 2.6.38.8 doesn't seem to have this patch yet.

I had a single Mantis card and a single CPU.
I used VDR to deliver HDTV channel via network
into a faster computer for viewing.

One known latency point with Mantis is I2C transfer.
With my measure, latency is about 0.33ms per I2C transfer byte.
Basic read / write (address+value) minimal latency is thus 0.66ms.
Latency amount depends on I2C bus speed on the card.
I don't have a working patch to fix this yet though.

Regards, Marko Ristola

-----------------------------------------------------------------------

Refactor Mantis DMA transfer to deliver 16Kb TS data per interrupt


https://patchwork.kernel.org/patch/107036/

With VDR streaming HDTV into network, generating an interrupt once per 16kb,
implemented in this patch, seems to support more robust throughput with HDTV.

Fix leaking almost 64kb data from the previous TS after changing the TS.
One effect of the old version was, that the DMA transfer and driver's
DMA buffer access might happen at the same time - a race condition.

Signed-off-by: Marko M. Ristola <marko.ristola@kolumbus.fi>

Regards,
Marko Ristola

diff --git a/drivers/media/dvb/mantis/hopper_cards.c b/drivers/media/dvb/mantis/hopper_cards.c
index 09e9fc7..3b7e376 100644
--- a/drivers/media/dvb/mantis/hopper_cards.c
+++ b/drivers/media/dvb/mantis/hopper_cards.c
@@ -126,7 +126,7 @@ static irqreturn_t hopper_irq_handler(int irq, void *dev_id)
  	}
  	if (stat & MANTIS_INT_RISCI) {
  		dprintk(MANTIS_DEBUG, 0, "<%s>", label[8]);
-		mantis->finished_block = (stat & MANTIS_INT_RISCSTAT) >> 28;
+		mantis->busy_block = (stat & MANTIS_INT_RISCSTAT) >> 28;
  		tasklet_schedule(&mantis->tasklet);
  	}
  	if (stat & MANTIS_INT_I2CDONE) {
diff --git a/drivers/media/dvb/mantis/mantis_cards.c b/drivers/media/dvb/mantis/mantis_cards.c
index cf4b39f..8f048d5 100644
--- a/drivers/media/dvb/mantis/mantis_cards.c
+++ b/drivers/media/dvb/mantis/mantis_cards.c
@@ -134,7 +134,7 @@ static irqreturn_t mantis_irq_handler(int irq, void *dev_id)
  	}
  	if (stat & MANTIS_INT_RISCI) {
  		dprintk(MANTIS_DEBUG, 0, "<%s>", label[8]);
-		mantis->finished_block = (stat & MANTIS_INT_RISCSTAT) >> 28;
+		mantis->busy_block = (stat & MANTIS_INT_RISCSTAT) >> 28;
  		tasklet_schedule(&mantis->tasklet);
  	}
  	if (stat & MANTIS_INT_I2CDONE) {
diff --git a/drivers/media/dvb/mantis/mantis_common.h b/drivers/media/dvb/mantis/mantis_common.h
index d0b645a..23b23b7 100644
--- a/drivers/media/dvb/mantis/mantis_common.h
+++ b/drivers/media/dvb/mantis/mantis_common.h
@@ -122,11 +122,8 @@ struct mantis_pci {
  	unsigned int		num;
  
  	/*	RISC Core		*/
-	u32			finished_block;
+	u32			busy_block;
  	u32			last_block;
-	u32			line_bytes;
-	u32			line_count;
-	u32			risc_pos;
  	u8			*buf_cpu;
  	dma_addr_t		buf_dma;
  	u32			*risc_cpu;
diff --git a/drivers/media/dvb/mantis/mantis_dma.c b/drivers/media/dvb/mantis/mantis_dma.c
index 46202a4..c61ca7d 100644
--- a/drivers/media/dvb/mantis/mantis_dma.c
+++ b/drivers/media/dvb/mantis/mantis_dma.c
@@ -43,13 +43,17 @@
  #define RISC_IRQ		(0x01 << 24)
  
  #define RISC_STATUS(status)	((((~status) & 0x0f) << 20) | ((status & 0x0f) << 16))
-#define RISC_FLUSH()		(mantis->risc_pos = 0)
-#define RISC_INSTR(opcode)	(mantis->risc_cpu[mantis->risc_pos++] = cpu_to_le32(opcode))
+#define RISC_FLUSH(risc_pos)		(risc_pos = 0)
+#define RISC_INSTR(risc_pos, opcode)	(mantis->risc_cpu[risc_pos++] = cpu_to_le32(opcode))
  
  #define MANTIS_BUF_SIZE		(64 * 1024)
-#define MANTIS_BLOCK_BYTES	(MANTIS_BUF_SIZE >> 4)
-#define MANTIS_BLOCK_COUNT	(1 << 4)
-#define MANTIS_RISC_SIZE	PAGE_SIZE
+#define MANTIS_BLOCK_BYTES      (MANTIS_BUF_SIZE / 4)
+#define MANTIS_DMA_TR_BYTES     (2 * 1024) /* upper limit: 4095 bytes. */
+#define MANTIS_BLOCK_COUNT	(MANTIS_BUF_SIZE / MANTIS_BLOCK_BYTES)
+
+#define MANTIS_DMA_TR_UNITS     (MANTIS_BLOCK_BYTES / MANTIS_DMA_TR_BYTES)
+/* MANTIS_BUF_SIZE / MANTIS_DMA_TR_UNITS must not exceed MANTIS_RISC_SIZE (4k RISC cmd buffer) */
+#define MANTIS_RISC_SIZE	PAGE_SIZE /* RISC program must fit here. */
  
  int mantis_dma_exit(struct mantis_pci *mantis)
  {
@@ -124,27 +128,6 @@ err:
  	return -ENOMEM;
  }
  
-static inline int mantis_calc_lines(struct mantis_pci *mantis)
-{
-	mantis->line_bytes = MANTIS_BLOCK_BYTES;
-	mantis->line_count = MANTIS_BLOCK_COUNT;
-
-	while (mantis->line_bytes > 4095) {
-		mantis->line_bytes >>= 1;
-		mantis->line_count <<= 1;
-	}
-
-	dprintk(MANTIS_DEBUG, 1, "Mantis RISC block bytes=[%d], line bytes=[%d], line count=[%d]",
-		MANTIS_BLOCK_BYTES, mantis->line_bytes, mantis->line_count);
-
-	if (mantis->line_count > 255) {
-		dprintk(MANTIS_ERROR, 1, "Buffer size error");
-		return -EINVAL;
-	}
-
-	return 0;
-}
-
  int mantis_dma_init(struct mantis_pci *mantis)
  {
  	int err = 0;
@@ -158,12 +141,6 @@ int mantis_dma_init(struct mantis_pci *mantis)
  
  		goto err;
  	}
-	err = mantis_calc_lines(mantis);
-	if (err < 0) {
-		dprintk(MANTIS_ERROR, 1, "Mantis calc lines failed");
-
-		goto err;
-	}
  
  	return 0;
  err:
@@ -174,31 +151,32 @@ EXPORT_SYMBOL_GPL(mantis_dma_init);
  static inline void mantis_risc_program(struct mantis_pci *mantis)
  {
  	u32 buf_pos = 0;
-	u32 line;
+	u32 line, step;
+	u32 risc_pos;
  
  	dprintk(MANTIS_DEBUG, 1, "Mantis create RISC program");
-	RISC_FLUSH();
-
-	dprintk(MANTIS_DEBUG, 1, "risc len lines %u, bytes per line %u",
-		mantis->line_count, mantis->line_bytes);
-
-	for (line = 0; line < mantis->line_count; line++) {
-		dprintk(MANTIS_DEBUG, 1, "RISC PROG line=[%d]", line);
-		if (!(buf_pos % MANTIS_BLOCK_BYTES)) {
-			RISC_INSTR(RISC_WRITE	|
-				   RISC_IRQ	|
-				   RISC_STATUS(((buf_pos / MANTIS_BLOCK_BYTES) +
-				   (MANTIS_BLOCK_COUNT - 1)) %
-				    MANTIS_BLOCK_COUNT) |
-				    mantis->line_bytes);
-		} else {
-			RISC_INSTR(RISC_WRITE	| mantis->line_bytes);
-		}
-		RISC_INSTR(mantis->buf_dma + buf_pos);
-		buf_pos += mantis->line_bytes;
+	RISC_FLUSH(risc_pos);
+
+	dprintk(MANTIS_DEBUG, 1, "risc len lines %u, bytes per line %u, bytes per DMA tr %u",
+		MANTIS_BLOCK_COUNT, MANTIS_BLOCK_BYTES, MANTIS_DMA_TR_BYTES);
+
+	for (line = 0; line < MANTIS_BLOCK_COUNT; line++) {
+		for (step = 0; step < MANTIS_DMA_TR_UNITS; step++) {
+			dprintk(MANTIS_DEBUG, 1, "RISC PROG line=[%d], step=[%d]", line, step);
+			if (step == 0) {
+				RISC_INSTR(risc_pos, RISC_WRITE	|
+					   RISC_IRQ	|
+					   RISC_STATUS(line) |
+					   MANTIS_DMA_TR_BYTES);
+			} else {
+				RISC_INSTR(risc_pos, RISC_WRITE | MANTIS_DMA_TR_BYTES);
+			}
+			RISC_INSTR(risc_pos, mantis->buf_dma + buf_pos);
+			buf_pos += MANTIS_DMA_TR_BYTES;
+		  }
  	}
-	RISC_INSTR(RISC_JUMP);
-	RISC_INSTR(mantis->risc_dma);
+	RISC_INSTR(risc_pos, RISC_JUMP);
+	RISC_INSTR(risc_pos, mantis->risc_dma);
  }
  
  void mantis_dma_start(struct mantis_pci *mantis)
@@ -210,7 +188,7 @@ void mantis_dma_start(struct mantis_pci *mantis)
  	mmwrite(mmread(MANTIS_GPIF_ADDR) | MANTIS_GPIF_HIFRDWRN, MANTIS_GPIF_ADDR);
  
  	mmwrite(0, MANTIS_DMA_CTL);
-	mantis->last_block = mantis->finished_block = 0;
+	mantis->last_block = mantis->busy_block = 0;
  
  	mmwrite(mmread(MANTIS_INT_MASK) | MANTIS_INT_RISCI, MANTIS_INT_MASK);
  
@@ -245,9 +223,9 @@ void mantis_dma_xfer(unsigned long data)
  	struct mantis_pci *mantis = (struct mantis_pci *) data;
  	struct mantis_hwconfig *config = mantis->hwconfig;
  
-	while (mantis->last_block != mantis->finished_block) {
+	while (mantis->last_block != mantis->busy_block) {
  		dprintk(MANTIS_DEBUG, 1, "last block=[%d] finished block=[%d]",
-			mantis->last_block, mantis->finished_block);
+			mantis->last_block, mantis->busy_block);
  
  		(config->ts_size ? dvb_dmx_swfilter_204 : dvb_dmx_swfilter)
  		(&mantis->demux, &mantis->buf_cpu[mantis->last_block * MANTIS_BLOCK_BYTES], MANTIS_BLOCK_BYTES);

Re: Multiple Mantis devices gives me glitches

[Vidar Tyldum]

Marko Ristola, on 13.12.2011 08:31:
> Here is a patch that went into Linus GIT this year.
> It reduces the number of DMA transfer interrupts into one third.
> Linus released 2.6.38.8 doesn't seem to have this patch yet.

Thank you very much. I did see this patch mentioned in my quest for more knowledge, but was not sure of the status.

I have patched the Ubuntu 2.6.38.8 kernel for Natty and loaded the new driver. Works fine for one card at the moment, will insert the other two later this afternoon.
If this fixes my problem, how to proceed in getting the patch accepted as soon as possible? By me filing an official bug?

I'll do some stress testing by employing all adapters at once through VDR and look for any problems caused or fixed by the patch.

-- 
Vidar Tyldum
                              vidar@tyldum.com               PGP: 0x3110AA98

Re: Multiple Mantis devices gives me glitches

[Vidar Tyldum]

13.12.2011 08:31, Marko Ristola:
> 
> Hi
> 
> Here is a patch that went into Linus GIT this year.
> It reduces the number of DMA transfer interrupts into one third.
> Linus released 2.6.38.8 doesn't seem to have this patch yet

Good news, combining this patch with IRQ management fixes the problem for me.

Status:
 * Stock 2.6.38.8 mantis, no IRQ management:     glitches
 * Stock 2.6.38.8 mantis, with IRQ management:   glitches
 * Patched 2.6.38.8 mantis, no IRQ management:   glitches (less)
 * Patched 2.6.38.8 mantis, with IRQ management: very few glitches
 * Same as above, but latency_timer set to 0xff: no glitches in one hour

The patch was applied to 2.6.38.8 (in Ubuntu terms: 2.6.38-13-generic-pae).
Tests involved having VDR record on three different transponders at the same
time, which means lots of IO and all three cards active at once.

For IRQ management I tried both 'irqbalancer' and manual setting with IRQ
affinity (which is basicly what irqbalancer does).

I tried playing with the latency timer on the other scenarios, not only the
last one, but all had glitches anyways.

Not sure what to conclude with:
 - Unfortunate IRQ handling on this CPU (two and two share IRQ handling)?
 - Too many interrupts generated by driver (design issues)?
 - Driver not handling SMP gracefully?
...or a combination?

Your patch is definately needed. Is there anything I can do to help getting
it in?


-- 
Vidar Tyldum
                              vidar@tyldum.com               PGP: 0x3110AA98

Re: Multiple Mantis devices gives me glitches

[Marko Ristola]

13.12.2011 20:11, Vidar Tyldum kirjoitti:
> 13.12.2011 08:31, Marko Ristola:
>>
>> Hi
>>
>> Here is a patch that went into Linus GIT this year.
>> It reduces the number of DMA transfer interrupts into one third.
>> Linus released 2.6.38.8 doesn't seem to have this patch yet
>
> Good news, combining this patch with IRQ management fixes the problem for me.
>
> Status:
>   * Stock 2.6.38.8 mantis, no IRQ management:     glitches
>   * Stock 2.6.38.8 mantis, with IRQ management:   glitches
>   * Patched 2.6.38.8 mantis, no IRQ management:   glitches (less)
>   * Patched 2.6.38.8 mantis, with IRQ management: very few glitches
>   * Same as above, but latency_timer set to 0xff: no glitches in one hour
>
> The patch was applied to 2.6.38.8 (in Ubuntu terms: 2.6.38-13-generic-pae).
> Tests involved having VDR record on three different transponders at the same
> time, which means lots of IO and all three cards active at once.
>
> For IRQ management I tried both 'irqbalancer' and manual setting with IRQ
> affinity (which is basicly what irqbalancer does).
>
> I tried playing with the latency timer on the other scenarios, not only the
> last one, but all had glitches anyways.
>
> Not sure what to conclude with:
>   - Unfortunate IRQ handling on this CPU (two and two share IRQ handling)?
>   - Too many interrupts generated by driver (design issues)?
>   - Driver not handling SMP gracefully?
> ...or a combination?
>
> Your patch is definately needed. Is there anything I can do to help getting
> it in?

Here is a reference for Linu's master GIT tree for
the patch that went in:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=79d06d4dff733ee472e1f8933a33317a18195c0c
It will be in Linux 3.2 kernel release.
Ubuntu will take it in some day when they grab a kernel with number at least 3.2.

If you describe your problem into Ubuntu bug report system with
"There is a fix already accepted upstream. Here is a reference for the patch.",
there is a chance that you get it backported too.

I tested the patch originally very thoroughly. It improves things, but doesn't break any.
VDR wants data from the DVB card in large chunks to avoid unnecessary CPU context switching.
DVB 16KB patch is designed so, that DVB card has always some data to be delivered for VDR,
but at the same time it avoids doing unnecessary many interrupts.

Here is another patch that I wrote for my DVB HDTV network delivery glitches problem.
It might help you a bit also:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=38e009aac9e02d2c30fd9a5e979ab31433e7d578
It tries to copy DMA buffer data as fast as possible into DVB-CORE's internal 128K buffer.
Thus it prevents from Mantis DMA 64K buffer from being overrun.
It helps lots if the original dvb_dmx_swfilter(_204) is very slow, which
I saw with "perf top" tool.
Later I detected that it helps with some computers a lot, and with some other computers the patch doesn't help so much.

That patch doesn't guarantee glitch free data delivery either.
Next question is, which buffer overruns now? Mantis 64K DMA buffer? DVB-CORE's 128K buffer?
VDR reads from DVB-CORE's 128K buffer.

dvb_dmx_swfilter(_204) in media/dvb-core/dvb_demux.c is designed to drop data if DVB-CORE's 128K buffer
becomes full.

Maybe we could add into mantis_dma.c some logic:
"Mantis DMA transfer delivers data into DVB-CORE's 128K buffer in 16K chunks, avoiding the 128K buffer overrun."
"If DVB-CORE's buffer would overrun, Mantis DMA transfer sends data, if also Mantis 64K DMA buffer is
becoming full.

I suspect I2C latency problem to be a root cause for these glitches,
but there isn't a patch to test it yet. I2C latency should not be a problem
with quad core CPU: it should affect most with a single CPU computer.
"perf top" shows easilly some CPU hogs.

Regards,
Marko Ristola

>
>

Re: Multiple Mantis devices gives me glitches

[Vidar Tyldum]

13.12.2011 22:56, Marko Ristola:
> Here is another patch that I wrote for my DVB HDTV network delivery glitches
> problem.
> It might help you a bit also:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=38e009aac9e02d2c30fd9a5e979ab31433e7d578

I started seeing glitches again after a day, so I decided to give this a go.
With 'perf top' I see that the activity dvb_dmxdev_init was somewhat
reduced, but I still have some glitches.

perf top:
615.00 16.4% dvb_dmxdev_init

380 irqs/sec  kernel:49.5%

> That patch doesn't guarantee glitch free data delivery either.
> Next question is, which buffer overruns now? Mantis 64K DMA buffer?
> DVB-CORE's 128K buffer?
> VDR reads from DVB-CORE's 128K buffer.

I'll keep these two patches running for a while and see how it goes. Things
are definately better, but eventually I get glitches.
I am unable to locate anything else on the system causing load, but of
course, that does not mean there isn't any.

Thanks for the help. I'm always open for testing new patches. Sadly I am not
capable of diving into the code to help.

-- 
Vidar Tyldum
                              vidar@tyldum.com               PGP: 0x3110AA98