* RT_PREEMPT on Raspberry PI 3 @ 2016-09-18 10:03 Giuliano Colla 2016-09-22 14:10 ` Sebastian Andrzej Siewior 2016-12-01 2:25 ` Trevor Woerner 0 siblings, 2 replies; 10+ messages in thread From: Giuliano Colla @ 2016-09-18 10:03 UTC (permalink / raw) To: linux-rt-users Hi everybody, I'm developing industrial applications requiring real-time performance on a Raspberry pi3 platform. I have managed to achieve both good performance and good stability, but I have been obliged to overcome some difficulties which IMHO could have been avoided if the Raspberry platform peculiarities had been given more attention. Being a very popular platform, particularly suited also for real-time applications, I believe it deserves it. The main items are: 1) The RT patches are not in sync with mainline Raspbian kernel versions. At the time of writing, the stock 4.x kernel supported is 4.4.21, while the last available patch is 4.4.19-rt27. Even going backwards it's impossible to find a perfect match: 4.4.14 against 4.4.12-rt20 or 4.4.15-rt23, and so on. I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was the best match I could find, and which required some minor manual adjustments. Is there a way to provide a better sync? 2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably frequent freezes. It appears to work properly only on the basic Raspberry pi (single core) but not on the multicore Pi2 and Pi3. Currently the only way to make a real-time kernel run on Raspberry PI 2 or 3 is to disable FIQ on the command line. If the RT_PREEMPT patch could somehow support this incompatibility (either by fully disabling FIQ or fixing the code) this would be a great help, thus avoiding populating forums with "my raspberry pi2/3 freezes with rt patch" Can something be done to solve those issues? Thanks, Giuliano ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-09-18 10:03 RT_PREEMPT on Raspberry PI 3 Giuliano Colla @ 2016-09-22 14:10 ` Sebastian Andrzej Siewior 2016-11-18 10:25 ` Oussama Ghorbel 2016-12-01 2:25 ` Trevor Woerner 1 sibling, 1 reply; 10+ messages in thread From: Sebastian Andrzej Siewior @ 2016-09-22 14:10 UTC (permalink / raw) To: Giuliano Colla; +Cc: linux-rt-users On 2016-09-18 12:03:42 [+0200], Giuliano Colla wrote: > I have managed to achieve both good performance and good stability, but I > have been obliged to overcome some difficulties which IMHO could have been > avoided if the Raspberry platform peculiarities had been given more > attention. Being a very popular platform, particularly suited also for > real-time applications, I believe it deserves it. who should give Raspberry more attention? > The main items are: > > 1) The RT patches are not in sync with mainline Raspbian kernel versions. At > the time of writing, the stock 4.x kernel supported is 4.4.21, while the > last available patch is 4.4.19-rt27. Even going backwards it's impossible to > find a perfect match: 4.4.14 against 4.4.12-rt20 or 4.4.15-rt23, and so on. we do have v4.4.21-rt30 as of the time of writing. I hope that you do not ask to align the -RT development & stable updates on the Raspberry tree. > I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was the > best match I could find, and which required some minor manual adjustments. > > Is there a way to provide a better sync? You can always take the latest -RT patch from the v4.4 branch and manually apply it on top of your BSP. And then fixup what broke. > 2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not > compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably > frequent freezes. It appears to work properly only on the basic Raspberry pi > (single core) but not on the multicore Pi2 and Pi3. Currently the only way > to make a real-time kernel run on Raspberry PI 2 or 3 is to disable FIQ on > the command line. I wouldn't say that this is PREEMPT-RT's fault. FIQ is a non-maskable interrupt which we do have on x86 and we call it NMI. As long as you don't do anything illegal like taking locks which may lead to a dead lock then no harm will happen to you. Therefore I assume that PREEMPT-RT triggers the bugs that are already there more reliably. > If the RT_PREEMPT patch could somehow support this incompatibility (either > by fully disabling FIQ or fixing the code) this would be a great help, thus > avoiding populating forums with "my raspberry pi2/3 freezes with rt patch" We could disable FIQ on rpi3 if PREEMPT_RT is enabled but this would require that the rpi3 is supported upstream. There is no way I suck the rpi3 bsp into the RT queue. > Can something be done to solve those issues? You could talk to the rpi3 people to put a link next to their bsp which describes the common pitfalls with -RT. > Thanks, > > Giuliano Sebastian ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-09-22 14:10 ` Sebastian Andrzej Siewior @ 2016-11-18 10:25 ` Oussama Ghorbel 2016-11-18 14:21 ` Jeff Epler ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Oussama Ghorbel @ 2016-11-18 10:25 UTC (permalink / raw) To: Giuliano Colla; +Cc: linux-rt-users, Sebastian Andrzej Siewior Hi, I have written recently a patch that solve the FIQ issue with RT Preempt on the 4-core Raspberry Pi board. Please see this article for full description on the solution. https://www.osadl.org/Single-View.111+M5c03315dc57.0.html Regards, Oussama On Thu, Sep 22, 2016 at 3:10 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > On 2016-09-18 12:03:42 [+0200], Giuliano Colla wrote: >> I have managed to achieve both good performance and good stability, but I >> have been obliged to overcome some difficulties which IMHO could have been >> avoided if the Raspberry platform peculiarities had been given more >> attention. Being a very popular platform, particularly suited also for >> real-time applications, I believe it deserves it. > > who should give Raspberry more attention? > >> The main items are: >> >> 1) The RT patches are not in sync with mainline Raspbian kernel versions. At >> the time of writing, the stock 4.x kernel supported is 4.4.21, while the >> last available patch is 4.4.19-rt27. Even going backwards it's impossible to >> find a perfect match: 4.4.14 against 4.4.12-rt20 or 4.4.15-rt23, and so on. > > we do have v4.4.21-rt30 as of the time of writing. I hope that you do > not ask to align the -RT development & stable updates on the Raspberry > tree. > >> I'm currently using a 4.4.13 kernel with a 4.4.12-rt19 patch, which was the >> best match I could find, and which required some minor manual adjustments. >> >> Is there a way to provide a better sync? > > You can always take the latest -RT patch from the v4.4 branch and > manually apply it on top of your BSP. And then fixup what broke. > >> 2) The RT_PREEMPT patch is not compatible with FIQ (or maybe FIQ is not >> compatible with RT_PREEMPT). Leaving FIQ enabled generates intolerably >> frequent freezes. It appears to work properly only on the basic Raspberry pi >> (single core) but not on the multicore Pi2 and Pi3. Currently the only way >> to make a real-time kernel run on Raspberry PI 2 or 3 is to disable FIQ on >> the command line. > > I wouldn't say that this is PREEMPT-RT's fault. FIQ is a non-maskable > interrupt which we do have on x86 and we call it NMI. As long as you > don't do anything illegal like taking locks which may lead to a dead > lock then no harm will happen to you. Therefore I assume that PREEMPT-RT > triggers the bugs that are already there more reliably. > >> If the RT_PREEMPT patch could somehow support this incompatibility (either >> by fully disabling FIQ or fixing the code) this would be a great help, thus >> avoiding populating forums with "my raspberry pi2/3 freezes with rt patch" > > We could disable FIQ on rpi3 if PREEMPT_RT is enabled but this would > require that the rpi3 is supported upstream. There is no way I suck the > rpi3 bsp into the RT queue. > >> Can something be done to solve those issues? > > You could talk to the rpi3 people to put a link next to their bsp which > describes the common pitfalls with -RT. > >> Thanks, >> >> Giuliano > > Sebastian > -- > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-11-18 10:25 ` Oussama Ghorbel @ 2016-11-18 14:21 ` Jeff Epler 2016-11-28 15:59 ` Sebastian Andrzej Siewior 2016-12-01 2:11 ` Trevor Woerner 2 siblings, 0 replies; 10+ messages in thread From: Jeff Epler @ 2016-11-18 14:21 UTC (permalink / raw) To: Oussama Ghorbel; +Cc: Giuliano Colla, linux-rt-users, Sebastian Andrzej Siewior FWIW the plain text version of the patch appears to be at https://www.osadl.org/monitoring/patches/rbs3s/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch I've included the current version of that patch inline below for reference. Jeff -- >8 -- From: Oussama Ghorbel <ghorbel@gmail.com> Date: Sun, 6 Nov 2016 00:16:02 +0100 Subject: [PATCH] usb: dwc_otg: fix system lockup when interrupts are threaded Fix lockup in dwc_otg driver that leads to a system freeze of the 4-core Raspberry Pi board when RT Preempt kernel is in use or when interrupts are threaded in general. The lockup occurs when the irq handler thread gets preempted while it holds the fiq spin lock. The patch makes sure to disable local irq while fiq spin lock is held irrespective of whether the interrupt is threaded or not. The patch also unifies the use of the fiq spin lock outside the fiq handler by introducing two function-like macros fiq_fsm_spin_lock_irqsave and fiq_fsm_spin_unlock_irqrestore. Under RT kernel, the bug can be reproduced in a few minutes by running hackbench and cyclictest in this way $ ( while true; do nice hackbench 30 >/dev/null; done )& $ echo "run 'kill $!' to stop hackbench" $ cyclictest -a -t -n -p 80 Signed-off-by: Oussama Ghorbel <ghorbel@gmail.com> --- drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h | 14 +++++++++ drivers/usb/host/dwc_otg/dwc_otg_hcd.c | 40 +++++++++++---------------- drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c | 39 ++++++++++---------------- drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c | 14 ++++----- 4 files changed, 52 insertions(+), 55 deletions(-) Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h =================================================================== --- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h +++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_fiq_fsm.h @@ -373,4 +373,18 @@ extern void dwc_otg_fiq_fsm(struct fiq_s extern void dwc_otg_fiq_nop(struct fiq_state *state); +#define fiq_fsm_spin_lock_irqsave(lock, flags) \ + do { \ + local_fiq_disable(); \ + local_irq_save(flags); \ + fiq_fsm_spin_lock(lock); \ + } while (0) + +#define fiq_fsm_spin_unlock_irqrestore(lock, flags) \ + do { \ + fiq_fsm_spin_unlock(lock); \ + local_irq_restore(flags); \ + local_fiq_enable(); \ + } while (0) + #endif /* DWC_OTG_FIQ_FSM_H_ */ Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd.c =================================================================== --- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd.c +++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd.c @@ -1418,12 +1418,11 @@ static void assign_and_init_hc(dwc_otg_h dwc_otg_hc_init(hcd->core_if, hc); - local_irq_save(flags); - if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); - } + if (fiq_enable) + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); + else + local_irq_save(flags); /* Enable the top level host channel interrupt. */ intr_enable = (1 << hc->hc_num); @@ -1433,12 +1432,10 @@ static void assign_and_init_hc(dwc_otg_h gintmsk.b.hcintr = 1; DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32); - if (fiq_enable) { - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); - } - - local_irq_restore(flags); + if (fiq_enable) + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); + else + local_irq_restore(flags); hc->qh = qh; } @@ -1616,6 +1613,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o int xfer_len, nrpackets; hcdma_data_t hcdma; hfnum_data_t hfnum; + unsigned long flags; if (st->fsm != FIQ_PASSTHROUGH) return 0; @@ -1691,8 +1689,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o fiq_print(FIQDBG_INT, hcd->fiq_state, "%08x", st->hctsiz_copy.d32); fiq_print(FIQDBG_INT, hcd->fiq_state, "%08x", st->hcdma_copy.d32); hfnum.d32 = DWC_READ_REG32(&hcd->core_if->host_if->host_global_regs->hfnum); - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_WRITE_REG32(&hc_regs->hctsiz, st->hctsiz_copy.d32); DWC_WRITE_REG32(&hc_regs->hcsplt, st->hcsplt_copy.d32); DWC_WRITE_REG32(&hc_regs->hcdma, st->hcdma_copy.d32); @@ -1712,8 +1709,7 @@ int fiq_fsm_queue_isoc_transaction(dwc_o } mb(); st->hcchar_copy.b.chen = 0; - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); return 0; } @@ -1739,6 +1735,7 @@ int fiq_fsm_queue_split_transaction(dwc_ /* Program HC registers, setup FIQ_state, examine FIQ if periodic, start transfer (not if uframe 5) */ int hub_addr, port_addr, frame, uframe; struct fiq_channel_state *st = &hcd->fiq_state->channel[hc->hc_num]; + unsigned long flags; if (st->fsm != FIQ_PASSTHROUGH) return 0; @@ -1847,8 +1844,7 @@ int fiq_fsm_queue_split_transaction(dwc_ DWC_WRITE_REG32(&hc_regs->hcchar, st->hcchar_copy.d32); DWC_WRITE_REG32(&hc_regs->hcintmsk, st->hcintmsk_copy.d32); - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); if (hc->ep_type & 0x1) { hfnum.d32 = DWC_READ_REG32(&hcd->core_if->host_if->host_global_regs->hfnum); @@ -1947,8 +1943,7 @@ int fiq_fsm_queue_split_transaction(dwc_ DWC_WRITE_REG32(&hc_regs->hcchar, st->hcchar_copy.d32); } mb(); - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); return 0; } @@ -2442,6 +2437,7 @@ static void process_non_periodic_channel void dwc_otg_hcd_queue_transactions(dwc_otg_hcd_t * hcd, dwc_otg_transaction_type_e tr_type) { + unsigned long flags; #ifdef DEBUG_SOF DWC_DEBUGPL(DBG_HCD, "Queue Transactions\n"); #endif @@ -2467,11 +2463,9 @@ void dwc_otg_hcd_queue_transactions(dwc_ gintmsk.b.nptxfempty = 1; if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, gintmsk.d32, 0); - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, gintmsk.d32, 0); } Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c =================================================================== --- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c +++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_intr.c @@ -107,6 +107,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_ gintmsk_data_t gintmsk; hfnum_data_t hfnum; haintmsk_data_t haintmsk; + unsigned long flags; #ifdef DEBUG dwc_otg_core_global_regs_t *global_regs = core_if->core_global_regs; @@ -124,8 +125,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_ /* Check if HOST Mode */ if (dwc_otg_is_host_mode(core_if)) { if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags); /* Pull in from the FIQ's disabled mask */ gintmsk.d32 = gintmsk.d32 | ~(dwc_otg_hcd->fiq_state->gintmsk_saved.d32); dwc_otg_hcd->fiq_state->gintmsk_saved.d32 = ~0; @@ -144,8 +144,7 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_ gintsts.d32 &= gintmsk.d32; if (fiq_enable) { - fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags); } if (!gintsts.d32) { @@ -192,11 +191,9 @@ int32_t dwc_otg_hcd_handle_intr(dwc_otg_ gintmsk_data_t gintmsk = { .b.portintr = 1}; retval |= dwc_otg_hcd_handle_port_intr(dwc_otg_hcd); if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&dwc_otg_hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32); - fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&dwc_otg_hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32); } @@ -236,8 +233,7 @@ exit_handler_routine: if (fiq_enable) { gintmsk_data_t gintmsk_new; haintmsk_data_t haintmsk_new; - local_fiq_disable(); - fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags); gintmsk_new.d32 = *(volatile uint32_t *)&dwc_otg_hcd->fiq_state->gintmsk_saved.d32; if(fiq_fsm_enable) haintmsk_new.d32 = *(volatile uint32_t *)&dwc_otg_hcd->fiq_state->haintmsk_saved.d32; @@ -260,8 +256,7 @@ exit_handler_routine: haintmsk.d32 = DWC_READ_REG32(&core_if->host_if->host_global_regs->haintmsk); /* Re-enable interrupts that the FIQ masked (first time round) */ FIQ_WRITE(dwc_otg_hcd->fiq_state->dwc_regs_base + GINTMSK, gintmsk.d32); - fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags); if ((jiffies / HZ) > last_time) { //dwc_otg_qh_t *qh; @@ -659,6 +654,7 @@ int32_t dwc_otg_hcd_handle_hc_intr(dwc_o { int i; int retval = 0; + unsigned long flags; haint_data_t haint = { .d32 = 0 } ; /* Clear appropriate bits in HCINTn to clear the interrupt bit in @@ -671,12 +667,10 @@ int32_t dwc_otg_hcd_handle_hc_intr(dwc_o if(fiq_fsm_enable) { /* check the mask? */ - local_fiq_disable(); - fiq_fsm_spin_lock(&dwc_otg_hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&dwc_otg_hcd->fiq_state->lock, flags); haint.b2.chint |= ~(dwc_otg_hcd->fiq_state->haintmsk_saved.b2.chint); dwc_otg_hcd->fiq_state->haintmsk_saved.b2.chint = ~0; - fiq_fsm_spin_unlock(&dwc_otg_hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&dwc_otg_hcd->fiq_state->lock, flags); } for (i = 0; i < dwc_otg_hcd->core_if->core_params->host_channels; i++) { @@ -1088,6 +1082,7 @@ static void halt_channel(dwc_otg_hcd_t * dwc_hc_t * hc, dwc_otg_qtd_t * qtd, dwc_otg_halt_status_e halt_status) { + unsigned long flags; if (hcd->core_if->dma_enable) { release_channel(hcd, hc, qtd, halt_status); return; @@ -1110,11 +1105,9 @@ static void halt_channel(dwc_otg_hcd_t * */ gintmsk.b.nptxfempty = 1; if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32); - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32); } @@ -1135,11 +1128,9 @@ static void halt_channel(dwc_otg_hcd_t * */ gintmsk.b.ptxfempty = 1; if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32); - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&global_regs->gintmsk, 0, gintmsk.d32); } Index: linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c =================================================================== --- linux-4.6.5-rt10-v7.orig/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c +++ linux-4.6.5-rt10-v7/drivers/usb/host/dwc_otg/dwc_otg_hcd_queue.c @@ -671,6 +671,7 @@ static int schedule_periodic(dwc_otg_hcd int dwc_otg_hcd_qh_add(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh) { int status = 0; + unsigned long flags; gintmsk_data_t intr_mask = {.d32 = 0 }; if (!DWC_LIST_EMPTY(&qh->qh_list_entry)) { @@ -689,11 +690,9 @@ int dwc_otg_hcd_qh_add(dwc_otg_hcd_t * h if ( !hcd->periodic_qh_count ) { intr_mask.b.sofintr = 1; if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, intr_mask.d32); - fiq_fsm_spin_unlock(&hcd->fiq 100 13425 100 13425 0 0 20549 0 --:--:-- --:--:-- --:--:-- 20527 _state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, intr_mask.d32); } @@ -737,6 +736,7 @@ static void deschedule_periodic(dwc_otg_ * @param qh QH to remove from schedule. */ void dwc_otg_hcd_qh_remove(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh) { + unsigned long flags; gintmsk_data_t intr_mask = {.d32 = 0 }; if (DWC_LIST_EMPTY(&qh->qh_list_entry)) { @@ -758,11 +758,9 @@ void dwc_otg_hcd_qh_remove(dwc_otg_hcd_t if( !hcd->periodic_qh_count && !fiq_fsm_enable ) { intr_mask.b.sofintr = 1; if (fiq_enable) { - local_fiq_disable(); - fiq_fsm_spin_lock(&hcd->fiq_state->lock); + fiq_fsm_spin_lock_irqsave(&hcd->fiq_state->lock, flags); DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, 0); - fiq_fsm_spin_unlock(&hcd->fiq_state->lock); - local_fiq_enable(); + fiq_fsm_spin_unlock_irqrestore(&hcd->fiq_state->lock, flags); } else { DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, intr_mask.d32, 0); } ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-11-18 10:25 ` Oussama Ghorbel 2016-11-18 14:21 ` Jeff Epler @ 2016-11-28 15:59 ` Sebastian Andrzej Siewior 2016-11-29 15:24 ` Oussama Ghorbel 2016-12-01 2:11 ` Trevor Woerner 2 siblings, 1 reply; 10+ messages in thread From: Sebastian Andrzej Siewior @ 2016-11-28 15:59 UTC (permalink / raw) To: Oussama Ghorbel; +Cc: Giuliano Colla, linux-rt-users On 2016-11-18 11:25:26 [+0100], Oussama Ghorbel wrote: > Hi, Hi, > I have written recently a patch that solve the FIQ issue with RT > Preempt on the 4-core Raspberry Pi board. > Please see this article for full description on the solution. > https://www.osadl.org/Single-View.111+M5c03315dc57.0.html So you have ~13ms max latency without the patch but with disabled FIQ and 170us with the Patch? Something does not look right. Anyway. If you don't mind I added this information to the RT-wiki [0]. So lets hope google will point RPI users there. Could please cross check if the information is correct and add other RPI informations? I named it "Raspberry PI 3". I am not sure if the problem is also relevant on "Raspberry PI" and "Raspberry PI 2". One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2 driver mainline. Is it possible to use that one instead the dwc_otg? It provides host and device functionality and it should be the same USB core. [0] https://wiki.linuxfoundation.org/realtime/documentation/known_limitations > Regards, > Oussama Sebastian ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-11-28 15:59 ` Sebastian Andrzej Siewior @ 2016-11-29 15:24 ` Oussama Ghorbel 2016-12-01 15:16 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 10+ messages in thread From: Oussama Ghorbel @ 2016-11-29 15:24 UTC (permalink / raw) To: Sebastian Andrzej Siewior; +Cc: Giuliano Colla, linux-rt-users >> I have written recently a patch that solve the FIQ issue with RT >> Preempt on the 4-core Raspberry Pi board. >> Please see this article for full description on the solution. >> https://www.osadl.org/Single-View.111+M5c03315dc57.0.html > > So you have ~13ms max latency without the patch but with disabled FIQ > and 170us with the Patch? Something does not look right. You're right. The 13 ms is not normal and that's because the primary system in rack #b, slot #3 was temporarily set back to a non-RT kernel for testing purposes (this is the one with FIQ disabled), but the shadow system always was running the RT kernel and always had the expected low latency. > Anyway. If you don't mind I added this information to the RT-wiki [0]. > So lets hope google will point RPI users there. Thanks for that. > Could please cross check if the information is correct and add other RPI > informations? I named it "Raspberry PI 3". I am not sure if the problem > is also relevant on "Raspberry PI" and "Raspberry PI 2". The problem is in fact relevant to the 4-core version of RPi, and this include the RPi2 and RPi 3. The original RPi was UP system and doesn't suffer from this issue. The patch however works on both of them (2 and 3) as can be seen in the RPi 3 in the shadow system of of rack #7, slot #3. You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3", otherwise the information you've written is correct. > One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2 > driver mainline. Is it possible to use that one instead the dwc_otg? It > provides host and device functionality and it should be the same USB > core. According to dwc2 linux documentation, this driver is compatible with brcm,bcm2835-usb, so I think it's probably possible to do it. However dwc2 driver doesn't use FIQ at all, and therefore you don't need the patch for it. By the way I have written two patches that enable to see a counter for the FIQ. https://www.osadl.org/monitoring/patches/r7s3s/0001-arm-add-stat-support-to-fiq.patch https://www.osadl.org/monitoring/patches/r7s3s/0002-usb-dwc_otg-enable-fiq-stat.patch You may see the output of /proc/interrupts on OSADL QA Farm on the shadow system of r7s3 and rbs3. > [0] https://wiki.linuxfoundation.org/realtime/documentation/known_limitations Oussama ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-11-29 15:24 ` Oussama Ghorbel @ 2016-12-01 15:16 ` Sebastian Andrzej Siewior 2016-12-01 16:24 ` Oussama Ghorbel 0 siblings, 1 reply; 10+ messages in thread From: Sebastian Andrzej Siewior @ 2016-12-01 15:16 UTC (permalink / raw) To: Oussama Ghorbel; +Cc: Giuliano Colla, linux-rt-users On 2016-11-29 16:24:08 [+0100], Oussama Ghorbel wrote: > > Could please cross check if the information is correct and add other RPI > > informations? I named it "Raspberry PI 3". I am not sure if the problem > > is also relevant on "Raspberry PI" and "Raspberry PI 2". > The problem is in fact relevant to the 4-core version of RPi, and this > include the RPi2 and RPi 3. The original RPi was UP system and doesn't > suffer from this issue. > The patch however works on both of them (2 and 3) as can be seen in > the RPi 3 in the shadow system of of rack #7, slot #3. > You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3", > otherwise the information you've written is correct. Updated. What is the benefit of using the FIQ? From maintenance point of view I would say it is easier to tell modprobe to disable FIQ instead to carry that patch. > > One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2 > > driver mainline. Is it possible to use that one instead the dwc_otg? It > > provides host and device functionality and it should be the same USB > > core. > According to dwc2 linux documentation, this driver is compatible with > brcm,bcm2835-usb, so I think it's probably possible to do it. > However dwc2 driver doesn't use FIQ at all, and therefore you don't > need the patch for it. Well, yes :) If it could be confirmed that this (in-tree) driver is working then I would add a pointer to the wiki to stay with the in-tree driver for USB instead using the OOT code. The v4.8 tree has a dts for RPI2 so that one could work without an OOT patch. For RPI3 it seems, that there is no in-tree support available. > Oussama Sebastian ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-12-01 15:16 ` Sebastian Andrzej Siewior @ 2016-12-01 16:24 ` Oussama Ghorbel 0 siblings, 0 replies; 10+ messages in thread From: Oussama Ghorbel @ 2016-12-01 16:24 UTC (permalink / raw) To: Sebastian Andrzej Siewior; +Cc: Giuliano Colla, linux-rt-users >> > Could please cross check if the information is correct and add other RPI >> > informations? I named it "Raspberry PI 3". I am not sure if the problem >> > is also relevant on "Raspberry PI" and "Raspberry PI 2". >> The problem is in fact relevant to the 4-core version of RPi, and this >> include the RPi2 and RPi 3. The original RPi was UP system and doesn't >> suffer from this issue. >> The patch however works on both of them (2 and 3) as can be seen in >> the RPi 3 in the shadow system of of rack #7, slot #3. >> You may modify the title from "Raspberry PI 3" to "Raspberry PI 2/3", >> otherwise the information you've written is correct. > > Updated. What is the benefit of using the FIQ? From maintenance point of > view I would say it is easier to tell modprobe to disable FIQ instead to > carry that patch. The FIQ will give us better determinism, better latency as It can be seen on OSADL QA Farm. >From today's test we have 242us (FIQ disabled) vs 172us (with patch) Moreover you will loose some features when disable FIQ. As FIQ is used as finite sate machine for very low level function like usb split transaction which is useful when plug low/full speed device to high speed usb hub. >> > One thing: The patch is against dwc_otg. Since v3.14 we have the dwc2 >> > driver mainline. Is it possible to use that one instead the dwc_otg? It >> > provides host and device functionality and it should be the same USB >> > core. >> According to dwc2 linux documentation, this driver is compatible with >> brcm,bcm2835-usb, so I think it's probably possible to do it. >> However dwc2 driver doesn't use FIQ at all, and therefore you don't >> need the patch for it. > > Well, yes :) If it could be confirmed that this (in-tree) driver is > working then I would add a pointer to the wiki to stay with the in-tree > driver for USB instead using the OOT code. > The v4.8 tree has a dts for RPI2 so that one could work without an OOT > patch. For RPI3 it seems, that there is no in-tree support available. I made a quick check to use dwc2 driver, unfortunately the usb was not initialized properly. Oussama ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-11-18 10:25 ` Oussama Ghorbel 2016-11-18 14:21 ` Jeff Epler 2016-11-28 15:59 ` Sebastian Andrzej Siewior @ 2016-12-01 2:11 ` Trevor Woerner 2 siblings, 0 replies; 10+ messages in thread From: Trevor Woerner @ 2016-12-01 2:11 UTC (permalink / raw) To: Oussama Ghorbel; +Cc: Giuliano Colla, linux-rt-users, Sebastian Andrzej Siewior On Fri 2016-11-18 @ 11:25:26 AM, Oussama Ghorbel wrote: > I have written recently a patch that solve the FIQ issue with RT > Preempt on the 4-core Raspberry Pi board. > Please see this article for full description on the solution. > https://www.osadl.org/Single-View.111+M5c03315dc57.0.html This is excellent, thank you very much! Previously my raspi2/3 systems were barely able to survive 15 minutes of either "stress" or "cyclictest", but by simply adding the two kernel cmdline options I've now been running: $ stress --cpu 5 --io 4 --vm 2 --vm-bytes 128M& $ cyclictest -p 80 -t5 -n on a raspi3 for over 10 hours. The latest cyclictest numbers are: policy: fifo: loadavg: 11.72 11.87 11.85 9/180 3097 T: 0 ( 525) P:80 I:1000 C:37527692 Min: 12 Act: 21 Avg: 25 Max: 725 T: 1 ( 526) P:80 I:1500 C:25018460 Min: 13 Act: 20 Avg: 23 Max: 540 T: 2 ( 527) P:80 I:2000 C:18763840 Min: 13 Act: 23 Avg: 24 Max: 914 T: 3 ( 528) P:80 I:2500 C:15011069 Min: 14 Act: 19 Avg: 26 Max: 514 T: 4 ( 529) P:80 I:3000 C:12509225 Min: 14 Act: 19 Avg: 24 Max: 568 This is running the 4.4.35 linux-raspberrypi kernel from git://github.com/raspberrypi/linux.git with patch-4.4.32-rt43.patch.xz applied (only one small hunk had to be removed from the Makefile patch in order for it to apply cleanly). $ uname -a Linux raspberrypi3 4.4.35-rt43-yocto-standard #1 SMP PREEMPT RT Tue Nov 29 15:16:27 EST 2016 armv7l armv7l armv7l GNU/Linux Best regards, Trevor ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RT_PREEMPT on Raspberry PI 3 2016-09-18 10:03 RT_PREEMPT on Raspberry PI 3 Giuliano Colla 2016-09-22 14:10 ` Sebastian Andrzej Siewior @ 2016-12-01 2:25 ` Trevor Woerner 1 sibling, 0 replies; 10+ messages in thread From: Trevor Woerner @ 2016-12-01 2:25 UTC (permalink / raw) To: Giuliano Colla; +Cc: linux-rt-users On Sun 2016-09-18 @ 12:03:42 PM, Giuliano Colla wrote: > 1) The RT patches are not in sync with mainline Raspbian kernel versions. The PREEMPT_RT patches aren't only meant for RaspberryPi devices. But even if they were, there are dozens (!) of GNU/Linux-based distributions which target RaspberryPi devices[1]. I'm glad the -rt developers don't show bias in the distros to which they align their patches :-) [1] http://elinux.org/RPi_Distributions (NOTE: not all of the listed distros are GNU/Linux, but many of them are) ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-12-01 16:25 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-09-18 10:03 RT_PREEMPT on Raspberry PI 3 Giuliano Colla 2016-09-22 14:10 ` Sebastian Andrzej Siewior 2016-11-18 10:25 ` Oussama Ghorbel 2016-11-18 14:21 ` Jeff Epler 2016-11-28 15:59 ` Sebastian Andrzej Siewior 2016-11-29 15:24 ` Oussama Ghorbel 2016-12-01 15:16 ` Sebastian Andrzej Siewior 2016-12-01 16:24 ` Oussama Ghorbel 2016-12-01 2:11 ` Trevor Woerner 2016-12-01 2:25 ` Trevor Woerner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).