RT_PREEMPT on Raspberry PI 3

RT_PREEMPT on Raspberry PI 3

[Giuliano Colla]

Hi everybody,

I'm developing industrial applications requiring real-time performance 
on a Raspberry pi3 platform.

I have managed to achieve both good performance and good stability, but 
I have been obliged to overcome some difficulties which IMHO could have 
been avoided if the Raspberry platform peculiarities had been given more 
attention. Being a very popular platform, particularly suited also for 
real-time applications, I believe it deserves it.

The main items are:

1) The RT patches are not in sync with mainline Raspbian kernel 
versions. At the time of writing, the stock 4.x kernel supported is 
4.4.21, while the last available patch is 4.4.19-rt27. Even going 
backwards it's impossible to find a perfect match: 4.4.14 against 
4.4.12-rt20 or 4.4.15-rt23, and so on.

I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was 
the best match I could find, and which required some minor manual 
adjustments.

Is there a way to provide a better sync?

2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not 
compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably 
frequent freezes. It appears to work properly only on the basic 
Raspberry pi (single core) but not on the multicore Pi2 and Pi3. 
Currently the only way to make a real-time kernel run on Raspberry PI 2 
or 3 is to disable FIQ on the command line.

If the RT_PREEMPT patch could somehow support this incompatibility 
(either by fully disabling FIQ or fixing the code) this would be a great 
help, thus avoiding populating forums with "my raspberry pi2/3 freezes 
with rt patch"

Can something be done to solve those issues?

Thanks,

Giuliano

Re: RT_PREEMPT on Raspberry PI 3

[Sebastian Andrzej Siewior]

On 2016-09-18 12:03:42 [+0200], Giuliano Colla wrote:
> I have managed to achieve both good performance and good stability, but I
> have been obliged to overcome some difficulties which IMHO could have been
> avoided if the Raspberry platform peculiarities had been given more
> attention. Being a very popular platform, particularly suited also for
> real-time applications, I believe it deserves it.

who should give Raspberry more attention?

> The main items are:
> 
> 1) The RT patches are not in sync with mainline Raspbian kernel versions. At
> the time of writing, the stock 4.x kernel supported is 4.4.21, while the
> last available patch is 4.4.19-rt27. Even going backwards it's impossible to
> find a perfect match: 4.4.14 against 4.4.12-rt20 or 4.4.15-rt23, and so on.

we do have v4.4.21-rt30 as of the time of writing. I hope that you do
not ask to align the -RT development & stable updates on the Raspberry
tree.

> I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was the
> best match I could find, and which required some minor manual adjustments.
> 
> Is there a way to provide a better sync?

You can always take the latest -RT patch from the v4.4 branch and
manually apply it on top of your BSP. And then fixup what broke.

> 2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not
> compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably
> frequent freezes. It appears to work properly only on the basic Raspberry pi
> (single core) but not on the multicore Pi2 and Pi3. Currently the only way
> to make a real-time kernel run on Raspberry PI 2 or 3 is to disable FIQ on
> the command line.

I wouldn't say that this is PREEMPT-RT's fault. FIQ is a non-maskable
interrupt which we do have on x86 and we call it NMI. As long as you
don't do anything illegal like taking locks which may lead to a dead
lock then no harm will happen to you. Therefore I assume that PREEMPT-RT
triggers the bugs that are already there more reliably.

> If the RT_PREEMPT patch could somehow support this incompatibility (either
> by fully disabling FIQ or fixing the code) this would be a great help, thus
> avoiding populating forums with "my raspberry pi2/3 freezes with rt patch"

We could disable FIQ on rpi3 if PREEMPT_RT is enabled but this would
require that the rpi3 is supported upstream. There is no way I suck the
rpi3 bsp into the RT queue. 

> Can something be done to solve those issues?

You could talk to the rpi3 people to put a link next to their bsp which
describes the common pitfalls with -RT.

> Thanks,
> 
> Giuliano

Sebastian

Re: RT_PREEMPT on Raspberry PI 3

[Oussama Ghorbel]

Hi,

I have written recently a patch that solve the FIQ issue with RT
Preempt on the 4-core Raspberry Pi board.
Please see this article for full description on the solution.
https://www.osadl.org/Single-View.111+M5c03315dc57.0.html

Regards,
Oussama

On Thu, Sep 22, 2016 at 3:10 PM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> On 2016-09-18 12:03:42 [+0200], Giuliano Colla wrote:
>> I have managed to achieve both good performance and good stability, but I
>> have been obliged to overcome some difficulties which IMHO could have been
>> avoided if the Raspberry platform peculiarities had been given more
>> attention. Being a very popular platform, particularly suited also for
>> real-time applications, I believe it deserves it.
>
> who should give Raspberry more attention?
>
>> The main items are:
>>
>> 1) The RT patches are not in sync with mainline Raspbian kernel versions. At
>> the time of writing, the stock 4.x kernel supported is 4.4.21, while the
>> last available patch is 4.4.19-rt27. Even going backwards it's impossible to
>> find a perfect match: 4.4.14 against 4.4.12-rt20 or 4.4.15-rt23, and so on.
>
> we do have v4.4.21-rt30 as of the time of writing. I hope that you do
> not ask to align the -RT development & stable updates on the Raspberry
> tree.
>
>> I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was the
>> best match I could find, and which required some minor manual adjustments.
>>
>> Is there a way to provide a better sync?
>
> You can always take the latest -RT patch from the v4.4 branch and
> manually apply it on top of your BSP. And then fixup what broke.
>
>> 2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not
>> compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably
>> frequent freezes. It appears to work properly only on the basic Raspberry pi
>> (single core) but not on the multicore Pi2 and Pi3. Currently the only way
>> to make a real-time kernel run on Raspberry PI 2 or 3 is to disable FIQ on
>> the command line.
>
> I wouldn't say that this is PREEMPT-RT's fault. FIQ is a non-maskable
> interrupt which we do have on x86 and we call it NMI. As long as you
> don't do anything illegal like taking locks which may lead to a dead
> lock then no harm will happen to you. Therefore I assume that PREEMPT-RT
> triggers the bugs that are already there more reliably.
>
>> If the RT_PREEMPT patch could somehow support this incompatibility (either
>> by fully disabling FIQ or fixing the code) this would be a great help, thus
>> avoiding populating forums with "my raspberry pi2/3 freezes with rt patch"
>
> We could disable FIQ on rpi3 if PREEMPT_RT is enabled but this would
> require that the rpi3 is supported upstream. There is no way I suck the
> rpi3 bsp into the RT queue.
>
>> Can something be done to solve those issues?
>
> You could talk to the rpi3 people to put a link next to their bsp which
> describes the common pitfalls with -RT.
>
>> Thanks,
>>
>> Giuliano
>
> Sebastian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RT_PREEMPT on Raspberry PI 3

[Jeff Epler]

FWIW the plain text version of the patch appears to be at
https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch
I've included the current version of that patch inline below for reference.

Jeff

-- >8 --
From: Oussama Ghorbel <ghorbel@gmail.com>
Date: Sun, 6 Nov 2016 00:16:02 +0100
Subject: [PATCH] usb: dwc_otg: fix system lockup when interrupts are threaded

Fix lockup in dwc_otg driver that leads to a system freeze of the
4-core Raspberry Pi board when RT Preempt kernel is in use or when
interrupts are threaded in general.
The lockup occurs when the irq handler thread gets preempted while it
holds the fiq spin lock.
The patch makes sure to disable local irq while fiq spin lock is held
irrespective of whether the interrupt is threaded or not.
The patch also unifies the use of the fiq spin lock outside the fiq
handler by introducing two function-like macros fiq_fsm_spin_lock_irqsave
and fiq_fsm_spin_unlock_irqrestore.

Under RT kernel, the bug can be reproduced in a few minutes by running
hackbench and cyclictest in this way
$ ( while true; do nice hackbench 30 >/dev/null; done )&
$ echo "run 'kill $!' to stop hackbench"
$ cyclictest -a -t -n -p 80

Signed-off-by: Oussama Ghorbel <ghorbel@gmail.com>

---
 drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h   |   14 +++++++++
 drivers/usb/host/dwc_otg/dwc_otg_hcd.c       |   40 +++++++++++----------------
 drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c  |   39 ++++++++++----------------
 drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c |   14 ++++-----
 4 files changed, 52 insertions(+), 55 deletions(-)

Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h
===================================================================
--- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h
+++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h
@@ -373,4 +373,18 @@ extern void dwc_otg_fiq_fsm(struct fiq_s
 
 extern void dwc_otg_fiq_nop(struct fiq_state *state);
 
+#define fiq_fsm_spin_lock_irqsave(lock, flags)		\
+	do {						\
+		local_fiq_disable();			\
+		local_irq_save(flags);			\
+		fiq_fsm_spin_lock(lock);		\
+	} while (0)
+
+#define fiq_fsm_spin_unlock_irqrestore(lock, flags)	\
+	do {						\
+		fiq_fsm_spin_unlock(lock);		\
+		local_irq_restore(flags);		\
+		local_fiq_enable();			\
+	} while (0)
+
 #endif /* DWC_OTG_FIQ_FSM_H_ */
Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
===================================================================
--- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
+++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
@@ -1418,12 +1418,11 @@ static void assign_and_init_hc(dwc_otg_h
 
 	dwc_otg_hc_init(hcd->core_if, hc);
 
-	local_irq_save(flags);
 
-	if (fiq_enable) {
-		local_fiq_disable();
-		fiq_fsm_spin_lock(&hcd->fiq_state->lock);
-	}
+	if (fiq_enable)
+		fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
+	else
+		local_irq_save(flags);
 
 	/* Enable the top level host channel interrupt. */
 	intr_enable = (1 << hc->hc_num);
@@ -1433,12 +1432,10 @@ static void assign_and_init_hc(dwc_otg_h
 	gintmsk.b.hcintr = 1;
 	DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32);
 
-	if (fiq_enable) {
-		fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-		local_fiq_enable();
-	}
-	
-	local_irq_restore(flags);
+	if (fiq_enable)
+		fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
+	else
+		local_irq_restore(flags);
 	hc->qh = qh;
 }
 
@@ -1616,6 +1613,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o
 	int xfer_len, nrpackets;
 	hcdma_data_t hcdma;
 	hfnum_data_t hfnum;
+	unsigned long flags;
 
 	if (st->fsm != FIQ_PASSTHROUGH)
 		return 0;
@@ -1691,8 +1689,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o
 	fiq_print(FIQDBG_INT, hcd->fiq_state, "%08x", st->hctsiz_copy.d32);
 	fiq_print(FIQDBG_INT, hcd->fiq_state, "%08x", st->hcdma_copy.d32);
 	hfnum.d32 = DWC_READ_REG32(&hcd->core_if->host_if->host_global_regs->hfnum);
-	local_fiq_disable();
-	fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+	fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 	DWC_WRITE_REG32(&hc_regs->hctsiz, st->hctsiz_copy.d32);
 	DWC_WRITE_REG32(&hc_regs->hcsplt, st->hcsplt_copy.d32);
 	DWC_WRITE_REG32(&hc_regs->hcdma, st->hcdma_copy.d32);
@@ -1712,8 +1709,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o
 	}
 	mb();
 	st->hcchar_copy.b.chen = 0;
-	fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-	local_fiq_enable();
+	fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 	return 0;
 }
 
@@ -1739,6 +1735,7 @@ int fiq_fsm_queue_split_transaction(dwc_
 	/* Program HC registers, setup FIQ_state, examine FIQ if periodic, start transfer (not if uframe 5) */
 	int hub_addr, port_addr, frame, uframe;
 	struct fiq_channel_state *st = &hcd->fiq_state->channel[hc->hc_num];
+	unsigned long flags;
 
 	if (st->fsm != FIQ_PASSTHROUGH)
 		return 0;
@@ -1847,8 +1844,7 @@ int fiq_fsm_queue_split_transaction(dwc_
 	DWC_WRITE_REG32(&hc_regs->hcchar, st->hcchar_copy.d32);
 	DWC_WRITE_REG32(&hc_regs->hcintmsk, st->hcintmsk_copy.d32);
 
-	local_fiq_disable();
-	fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+	fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 
 	if (hc->ep_type & 0x1) {
 		hfnum.d32 = DWC_READ_REG32(&hcd->core_if->host_if->host_global_regs->hfnum);
@@ -1947,8 +1943,7 @@ int fiq_fsm_queue_split_transaction(dwc_
 		DWC_WRITE_REG32(&hc_regs->hcchar, st->hcchar_copy.d32);
 	}
 	mb();
-	fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-	local_fiq_enable();
+	fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 	return 0;
 }
 
@@ -2442,6 +2437,7 @@ static void process_non_periodic_channel
 void dwc_otg_hcd_queue_transactions(dwc_otg_hcd_t * hcd,
 				    dwc_otg_transaction_type_e tr_type)
 {
+	unsigned long flags;
 #ifdef DEBUG_SOF
 	DWC_DEBUGPL(DBG_HCD, "Queue Transactions\n");
 #endif
@@ -2467,11 +2463,9 @@ void dwc_otg_hcd_queue_transactions(dwc_
 			gintmsk.b.nptxfempty = 1;
 
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, gintmsk.d32, 0);
-				fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, gintmsk.d32, 0);
 			}
Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c
===================================================================
--- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c
+++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c
@@ -107,6 +107,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_
 	gintmsk_data_t gintmsk;
 	hfnum_data_t hfnum;
 	haintmsk_data_t haintmsk;
+	unsigned long flags;
 
 #ifdef DEBUG
 	dwc_otg_core_global_regs_t *global_regs = core_if->core_global_regs;
@@ -124,8 +125,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_
 	/* Check if HOST Mode */
 	if (dwc_otg_is_host_mode(core_if)) {
 		if (fiq_enable) {
-			local_fiq_disable();
-			fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock);
+			fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags);
 			/* Pull in from the FIQ's disabled mask */
 			gintmsk.d32 = gintmsk.d32 | ~(dwc_otg_hcd->fiq_state->gintmsk_saved.d32);
 			dwc_otg_hcd->fiq_state->gintmsk_saved.d32 = ~0;
@@ -144,8 +144,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_
 		gintsts.d32 &= gintmsk.d32;
 
 		if (fiq_enable) {
-			fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock);
-			local_fiq_enable();
+			fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags);
 		}
 
 		if (!gintsts.d32) {
@@ -192,11 +191,9 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_
 			gintmsk_data_t gintmsk = { .b.portintr = 1};
 			retval |= dwc_otg_hcd_handle_port_intr(dwc_otg_hcd);
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&dwc_otg_hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32);
-				fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&dwc_otg_hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32);
 			}
@@ -236,8 +233,7 @@ exit_handler_routine:
 	if (fiq_enable)	{
 		gintmsk_data_t gintmsk_new;
 		haintmsk_data_t haintmsk_new;
-		local_fiq_disable();
-		fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock);
+		fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags);
 		gintmsk_new.d32 = *(volatile uint32_t *)&dwc_otg_hcd->fiq_state->gintmsk_saved.d32;
 		if(fiq_fsm_enable)
 			haintmsk_new.d32 = *(volatile uint32_t *)&dwc_otg_hcd->fiq_state->haintmsk_saved.d32;
@@ -260,8 +256,7 @@ exit_handler_routine:
 		haintmsk.d32 = DWC_READ_REG32(&core_if->host_if->host_global_regs->haintmsk);
 		/* Re-enable interrupts that the FIQ masked (first time round) */
 		FIQ_WRITE(dwc_otg_hcd->fiq_state->dwc_regs_base + GINTMSK, gintmsk.d32);
-		fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock);
-		local_fiq_enable();
+		fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags);
 
 		if ((jiffies / HZ) > last_time) {
 			//dwc_otg_qh_t *qh;
@@ -659,6 +654,7 @@ int32_t dwc_otg_hcd_handle_hc_intr(dwc_o
 {
 	int i;
 	int retval = 0;
+	unsigned long flags;
 	haint_data_t haint = { .d32 = 0 } ;
 
 	/* Clear appropriate bits in HCINTn to clear the interrupt bit in
@@ -671,12 +667,10 @@ int32_t dwc_otg_hcd_handle_hc_intr(dwc_o
 	if(fiq_fsm_enable)
 	{
 		/* check the mask? */
-		local_fiq_disable();
-		fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock);
+		fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags);
 		haint.b2.chint |= ~(dwc_otg_hcd->fiq_state->haintmsk_saved.b2.chint);
 		dwc_otg_hcd->fiq_state->haintmsk_saved.b2.chint = ~0;
-		fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock);
-		local_fiq_enable();
+		fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags);
 	}
 
 	for (i = 0; i < dwc_otg_hcd->core_if->core_params->host_channels; i++) {
@@ -1088,6 +1082,7 @@ static void halt_channel(dwc_otg_hcd_t *
 			 dwc_hc_t * hc,
 			 dwc_otg_qtd_t * qtd, dwc_otg_halt_status_e halt_status)
 {
+	unsigned long flags;
 	if (hcd->core_if->dma_enable) {
 		release_channel(hcd, hc, qtd, halt_status);
 		return;
@@ -1110,11 +1105,9 @@ static void halt_channel(dwc_otg_hcd_t *
 			 */
 			gintmsk.b.nptxfempty = 1;
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32);
-				fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32);
 			}
@@ -1135,11 +1128,9 @@ static void halt_channel(dwc_otg_hcd_t *
 			 */
 			gintmsk.b.ptxfempty = 1;
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32);
-				fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32);
 			}
Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c
===================================================================
--- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c
+++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c
@@ -671,6 +671,7 @@ static int schedule_periodic(dwc_otg_hcd
 int dwc_otg_hcd_qh_add(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh)
 {
 	int status = 0;
+	unsigned long flags;
 	gintmsk_data_t intr_mask = {.d32 = 0 };
 
 	if (!DWC_LIST_EMPTY(&qh->qh_list_entry)) {
@@ -689,11 +690,9 @@ int dwc_otg_hcd_qh_add(dwc_otg_hcd_t * h
 		if ( !hcd->periodic_qh_count ) {
 			intr_mask.b.sofintr = 1;
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, intr_mask.d32);
-				fiq_fsm_spin_unlock(&hcd->fiq
100 13425  100 13425    0     0  20549      0 --:--:-- --:--:-- --:--:-- 20527
_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, intr_mask.d32);
 			}
@@ -737,6 +736,7 @@ static void deschedule_periodic(dwc_otg_
  * @param qh QH to remove from schedule. */
 void dwc_otg_hcd_qh_remove(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh)
 {
+	unsigned long flags;
 	gintmsk_data_t intr_mask = {.d32 = 0 };
 
 	if (DWC_LIST_EMPTY(&qh->qh_list_entry)) {
@@ -758,11 +758,9 @@ void dwc_otg_hcd_qh_remove(dwc_otg_hcd_t
 		if( !hcd->periodic_qh_count && !fiq_fsm_enable ) {
 			intr_mask.b.sofintr = 1;
 			if (fiq_enable) {
-				local_fiq_disable();
-				fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+				fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags);
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, 0);
-				fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-				local_fiq_enable();
+				fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags);
 			} else {
 				DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, 0);
 			}

Re: RT_PREEMPT on Raspberry PI 3

[Sebastian Andrzej Siewior]

On 2016-11-18 11:25:26 [+0100], Oussama Ghorbel wrote:
> Hi,
Hi,

> I have written recently a patch that solve the FIQ issue with RT
> Preempt on the 4-core Raspberry Pi board.
> Please see this article for full description on the solution.
> https://www.osadl.org/Single-View.111+M5c03315dc57.0.html

So you have ~13ms max latency without the patch but with disabled FIQ
and 170us with the Patch? Something does not look right.

Anyway. If you don't mind I added this information to the RT-wiki [0].
So lets hope google will point RPI users there.
Could please cross check if the information is correct and add other RPI
informations? I named it "Raspberry PI 3". I am not sure if the problem
is also relevant on "Raspberry PI" and "Raspberry PI 2".

One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2
driver mainline. Is it possible to use that one instead the dwc_otg? It
provides host and device functionality and it should be the same USB
core.

[0] https://wiki.linuxfoundation.org/realtime/documentation/known_limitations

> Regards,
> Oussama

Sebastian

Re: RT_PREEMPT on Raspberry PI 3

[Oussama Ghorbel]

>> I have written recently a patch that solve the FIQ issue with RT
>> Preempt on the 4-core Raspberry Pi board.
>> Please see this article for full description on the solution.
>> https://www.osadl.org/Single-View.111+M5c03315dc57.0.html
>
> So you have ~13ms max latency without the patch but with disabled FIQ
> and 170us with the Patch? Something does not look right.
You're right.  The 13 ms is not normal and that's because the primary
system in rack #b, slot #3 was temporarily set back to a non-RT kernel
for testing purposes (this is the one with FIQ disabled), but the
shadow system always was running the RT kernel and always had the
expected low latency.

> Anyway. If you don't mind I added this information to the RT-wiki [0].
> So lets hope google will point RPI users there.
Thanks for that.

> Could please cross check if the information is correct and add other RPI
> informations? I named it "Raspberry PI 3". I am not sure if the problem
> is also relevant on "Raspberry PI" and "Raspberry PI 2".
The problem is in fact relevant to the 4-core version of RPi, and this
include the RPi2 and RPi 3. The original RPi was UP system and doesn't
suffer from this issue.
The patch however works on both of them (2 and 3) as can be seen in
the RPi 3 in the shadow system of of rack #7, slot #3.
You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3",
otherwise the information you've written is correct.

> One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2
> driver mainline. Is it possible to use that one instead the dwc_otg? It
> provides host and device functionality and it should be the same USB
> core.
According to dwc2 linux documentation, this driver is compatible with
brcm,bcm2835-usb, so I think it's probably possible to do it.
However dwc2 driver doesn't use FIQ at all, and therefore you don't
need the patch for it.

By the way I have written two patches that enable to see a counter for the FIQ.
https://www.osadl.org/monitoring/patches/r7s3s/0001-arm-add-stat-support-to-fiq.patch
https://www.osadl.org/monitoring/patches/r7s3s/0002-usb-dwc_otg-enable-fiq-stat.patch
You may see the output of /proc/interrupts on OSADL QA Farm on the
shadow system of r7s3 and rbs3.

> [0] https://wiki.linuxfoundation.org/realtime/documentation/known_limitations

Oussama

Re: RT_PREEMPT on Raspberry PI 3

[Trevor Woerner]

On Fri 2016-11-18 @ 11:25:26 AM, Oussama Ghorbel wrote:
> I have written recently a patch that solve the FIQ issue with RT
> Preempt on the 4-core Raspberry Pi board.
> Please see this article for full description on the solution.
> https://www.osadl.org/Single-View.111+M5c03315dc57.0.html

This is excellent, thank you very much!

Previously my raspi2/3 systems were barely able to survive 15 minutes of
either "stress" or "cyclictest", but by simply adding the two kernel cmdline
options I've now been running:

	$ stress --cpu 5 --io 4 --vm 2 --vm-bytes 128M&
	$ cyclictest -p 80 -t5 -n

on a raspi3 for over 10 hours. The latest cyclictest numbers are:

	policy: fifo: loadavg: 11.72 11.87 11.85 9/180 3097           

	T: 0 (  525) P:80 I:1000 C:37527692 Min:     12 Act:   21 Avg:   25 Max:     725
	T: 1 (  526) P:80 I:1500 C:25018460 Min:     13 Act:   20 Avg:   23 Max:     540
	T: 2 (  527) P:80 I:2000 C:18763840 Min:     13 Act:   23 Avg:   24 Max:     914
	T: 3 (  528) P:80 I:2500 C:15011069 Min:     14 Act:   19 Avg:   26 Max:     514
	T: 4 (  529) P:80 I:3000 C:12509225 Min:     14 Act:   19 Avg:   24 Max:     568

This is running the 4.4.35 linux-raspberrypi kernel from
git://github.com/raspberrypi/linux.git with patch-4.4.32-rt43.patch.xz applied
(only one small hunk had to be removed from the Makefile patch in order for it
to apply cleanly).

$ uname -a
Linux raspberrypi3 4.4.35-rt43-yocto-standard #1 SMP PREEMPT RT Tue Nov 29 15:16:27 EST 2016 armv7l armv7l armv7l GNU/Linux

Best regards,
	Trevor

Re: RT_PREEMPT on Raspberry PI 3

[Trevor Woerner]

On Sun 2016-09-18 @ 12:03:42 PM, Giuliano Colla wrote:
> 1) The RT patches are not in sync with mainline Raspbian kernel versions.

The PREEMPT_RT patches aren't only meant for RaspberryPi devices. But even if
they were, there are dozens (!) of GNU/Linux-based distributions which target
RaspberryPi devices[1]. I'm glad the -rt developers don't show bias in the
distros to which they align their patches :-)




[1] http://elinux.org/RPi_Distributions (NOTE: not all of the listed distros
are GNU/Linux, but many of them are)

Re: RT_PREEMPT on Raspberry PI 3

[Sebastian Andrzej Siewior]

On 2016-11-29 16:24:08 [+0100], Oussama Ghorbel wrote:
> > Could please cross check if the information is correct and add other RPI
> > informations? I named it "Raspberry PI 3". I am not sure if the problem
> > is also relevant on "Raspberry PI" and "Raspberry PI 2".
> The problem is in fact relevant to the 4-core version of RPi, and this
> include the RPi2 and RPi 3. The original RPi was UP system and doesn't
> suffer from this issue.
> The patch however works on both of them (2 and 3) as can be seen in
> the RPi 3 in the shadow system of of rack #7, slot #3.
> You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3",
> otherwise the information you've written is correct.

Updated. What is the benefit of using the FIQ? From maintenance point of
view I would say it is easier to tell modprobe to disable FIQ instead to
carry that patch.

> > One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2
> > driver mainline. Is it possible to use that one instead the dwc_otg? It
> > provides host and device functionality and it should be the same USB
> > core.
> According to dwc2 linux documentation, this driver is compatible with
> brcm,bcm2835-usb, so I think it's probably possible to do it.
> However dwc2 driver doesn't use FIQ at all, and therefore you don't
> need the patch for it.

Well, yes :) If it could be confirmed that this (in-tree) driver is
working then I would add a pointer to the wiki to stay with the in-tree
driver for USB instead using the OOT code.
The v4.8 tree has a dts for RPI2 so that one could work without an OOT
patch. For RPI3 it seems, that there is no in-tree support available.

> Oussama

Sebastian

Re: RT_PREEMPT on Raspberry PI 3

[Oussama Ghorbel]

>> > Could please cross check if the information is correct and add other RPI
>> > informations? I named it "Raspberry PI 3". I am not sure if the problem
>> > is also relevant on "Raspberry PI" and "Raspberry PI 2".
>> The problem is in fact relevant to the 4-core version of RPi, and this
>> include the RPi2 and RPi 3. The original RPi was UP system and doesn't
>> suffer from this issue.
>> The patch however works on both of them (2 and 3) as can be seen in
>> the RPi 3 in the shadow system of of rack #7, slot #3.
>> You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3",
>> otherwise the information you've written is correct.
>
> Updated. What is the benefit of using the FIQ? From maintenance point of
> view I would say it is easier to tell modprobe to disable FIQ instead to
> carry that patch.
The FIQ will give us better determinism, better latency as It can be
seen on OSADL QA Farm.
>From today's test we have 242us (FIQ disabled) vs 172us (with patch)
Moreover you will loose some features when disable FIQ. As FIQ is used
as finite sate machine
for very low level function like usb split transaction which is useful when plug
low/full speed device to high speed usb hub.

>> > One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2
>> > driver mainline. Is it possible to use that one instead the dwc_otg? It
>> > provides host and device functionality and it should be the same USB
>> > core.
>> According to dwc2 linux documentation, this driver is compatible with
>> brcm,bcm2835-usb, so I think it's probably possible to do it.
>> However dwc2 driver doesn't use FIQ at all, and therefore you don't
>> need the patch for it.
>
> Well, yes :) If it could be confirmed that this (in-tree) driver is
> working then I would add a pointer to the wiki to stay with the in-tree
> driver for USB instead using the OOT code.
> The v4.8 tree has a dts for RPI2 so that one could work without an OOT
> patch. For RPI3 it seems, that there is no in-tree support available.
I made a quick check to use dwc2 driver, unfortunately the usb was not
initialized properly.

Oussama