Searching a Bug on Raspberry Pi

Searching a Bug on Raspberry Pi

[Georg Gast]

Hi there!

Currently i try to locate a bug in drivers/mmc/host/sdhci.c on 
RaspberryPi. I know it is not the mainline kernel...

Currently i use Kernel 3.8.13 and RT Patch 3.8.13-rt12 on the kernel 
found at git://github.com/raspberrypi/linux.git on branch rpi-3.8.y.

The bug shows up at booting and after 3 seconds of booting as it wants 
to mount the rootfs in a deadlock. as i compiled sdhci_bcm2708 as a 
module i can boot the kernel and system as i pushed the system to a usb 
stick. As far as good.

Now i try to find the bug, if the rootfs is on the sdcard. I narrowd the 
deadlock down to the following lines:

void sdhci_spin_lock_irqsave(struct sdhci_host *host,unsigned long *flags)
{
     return;
#ifdef CONFIG_PREEMPT
     if(enable_llm)
     {
         while(sdhci_locked)
         {
             preempt_schedule();
         }
         spin_lock_irqsave(&host->lock,*flags);
         disable_irq(host->irq);
         if(host->second_irq)
             disable_irq(host->second_irq);
         local_irq_enable();
     }
     else
#endif
         spin_lock_irqsave(&host->lock,*flags);
}

void sdhci_spin_unlock_irqrestore(struct sdhci_host *host,unsigned long 
flags)
{
     return;
#ifdef CONFIG_PREEMPT
     if(enable_llm)
     {
         local_irq_disable();
         if(host->second_irq)
             enable_irq(host->second_irq);
         enable_irq(host->irq);
     }
#endif
     spin_unlock_irqrestore(&host->lock,flags);
}

in drivers/mmc/host/sdhci.c . The both return statements are added by 
me. Now i can boot the RT Preempt kernel but it is for sure not the 
right fix.

My problem is, that i dont know how those both functions should work 
with the rt preempt patch.

What can you suggest me to fix that deadlock? I already bought the book 
"Linux Kernel Development" but i dont know how the kernel works exaktly 
with the PREEMPT patch. Can you point me to a site which shows how 
RTPREEMPT works in the irqsave/restore functions or can you please 
describe it to me?

Thank you
Georg

Re: Searching a Bug on Raspberry Pi

[Georg Gast]

Hi Tom,

the problem occurs even if i set enable_llm=0. The while loop is now not 
touched and the rpi deadlocks too.

As far as i understand it, the sdhci.c is the base for all sdhci card 
readers and the sdhci-*.c implement the ops for the specific hardware.

My problem is, that i dont have an other sdhci card reader here. It wont 
be possible to boot from, but i could try to read from it on a rt 
kernel. If i compile the sdhci_bcm2708 as a module, i could not get a 
device info from the rpi reader. Maybe an other reader works, then the 
real problem sits in sdhci_bcm2708.c (or at bad luck, it shows an bigger 
problem in sdhci.c).

Georg


Am 29.06.2013 23:36, schrieb Tom Cook:
> This is as far as I've got too.  I think you need to make the while loop
> conditional on CONFIG_PREEMPT_RT_FULL but I haven't had a chance to try it.
>
> Tom
>
> On 29 Jun 2013 21:16, "Georg Gast" <georg@schorsch-tech.de
> <mailto:georg@schorsch-tech.de>> wrote:
>
>     Hi there!
>
>     Currently i try to locate a bug in drivers/mmc/host/sdhci.c on
>     RaspberryPi. I know it is not the mainline kernel...
>
>     Currently i use Kernel 3.8.13 and RT Patch 3.8.13-rt12 on the kernel
>     found at git://github.com/raspberrypi/__linux.git
>     <http://github.com/raspberrypi/linux.git> on branch rpi-3.8.y.
>
>     The bug shows up at booting and after 3 seconds of booting as it
>     wants to mount the rootfs in a deadlock. as i compiled sdhci_bcm2708
>     as a module i can boot the kernel and system as i pushed the system
>     to a usb stick. As far as good.
>
>     Now i try to find the bug, if the rootfs is on the sdcard. I narrowd
>     the deadlock down to the following lines:
>
>     void sdhci_spin_lock_irqsave(struct sdhci_host *host,unsigned long
>     *flags)
>     {
>          return;
>     #ifdef CONFIG_PREEMPT
>          if(enable_llm)
>          {
>              while(sdhci_locked)
>              {
>                  preempt_schedule();
>              }
>              spin_lock_irqsave(&host->lock,__*flags);
>              disable_irq(host->irq);
>              if(host->second_irq)
>                  disable_irq(host->second_irq);
>              local_irq_enable();
>          }
>          else
>     #endif
>              spin_lock_irqsave(&host->lock,__*flags);
>     }
>
>     void sdhci_spin_unlock_irqrestore(__struct sdhci_host *host,unsigned
>     long flags)
>     {
>          return;
>     #ifdef CONFIG_PREEMPT
>          if(enable_llm)
>          {
>              local_irq_disable();
>              if(host->second_irq)
>                  enable_irq(host->second_irq);
>              enable_irq(host->irq);
>          }
>     #endif
>          spin_unlock_irqrestore(&host->__lock,flags);
>     }
>
>     in drivers/mmc/host/sdhci.c . The both return statements are added
>     by me. Now i can boot the RT Preempt kernel but it is for sure not
>     the right fix.
>
>     My problem is, that i dont know how those both functions should work
>     with the rt preempt patch.
>
>     What can you suggest me to fix that deadlock? I already bought the
>     book "Linux Kernel Development" but i dont know how the kernel works
>     exaktly with the PREEMPT patch. Can you point me to a site which
>     shows how RTPREEMPT works in the irqsave/restore functions or can
>     you please describe it to me?
>
>     Thank you
>     Georg
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-rt-users" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>     <http://vger.kernel.org/majordomo-info.html>
>

Re: Searching a Bug on Raspberry Pi

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

On 06/30/2013 06:42 AM, Georg Gast wrote:
> Hi Tom,
> 
> the problem occurs even if i set enable_llm=0. The while loop is now not
> touched and the rpi deadlocks too.
> 
> As far as i understand it, the sdhci.c is the base for all sdhci card
> readers and the sdhci-*.c implement the ops for the specific hardware.
> 
> My problem is, that i dont have an other sdhci card reader here. It wont
> be possible to boot from, but i could try to read from it on a rt
> kernel. If i compile the sdhci_bcm2708 as a module, i could not get a
> device info from the rpi reader. Maybe an other reader works, then the
> real problem sits in sdhci_bcm2708.c (or at bad luck, it shows an bigger
> problem in sdhci.c).
> 
> Georg
> 

Here's a quick and dirty hack to get things going.

Regards,

Jeremy

[-- Attachment #2: sdhci.c.diff --]
[-- Type: text/x-patch, Size: 1092 bytes --]

--- linux/drivers/mmc/host/sdhci.c	2013-07-03 11:54:06.306455988 +0200
+++ linux-rt/drivers/mmc/host/sdhci.c	2013-07-03 16:49:33.298601044 +0200
@@ -162,9 +162,17 @@
 			preempt_schedule();
 		}
 		spin_lock_irqsave(&host->lock,*flags);
+#ifdef CONFIG_PREEMPT_RT_FULL
+		disable_irq_nosync(host->irq);
+#else
 		disable_irq(host->irq);
+#endif
 		if(host->second_irq)
+#ifdef CONFIG_PREEMPT_RT_FULL
+			disable_irq_nosync(host->second_irq);
+#else
 			disable_irq(host->second_irq);
+#endif
 		local_irq_enable();
 	}
 	else
@@ -1834,7 +1842,11 @@
 	host = mmc_priv(mmc);
 
 	sdhci_runtime_pm_get(host);
+#ifdef CONFIG_PREEMPT_RT_FULL
+	disable_irq_nosync(host->irq);
+#else
 	disable_irq(host->irq);
+#endif
 	spin_lock(&host->lock);
 
 	ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2);
@@ -1935,7 +1947,11 @@
 		wait_event_interruptible_timeout(host->buf_ready_int,
 					(host->tuning_done == 1),
 					msecs_to_jiffies(50));
+#ifdef CONFIG_PREEMPT_RT_FULL
+		disable_irq_nosync(host->irq);
+#else
 		disable_irq(host->irq);
+#endif
 		spin_lock(&host->lock);
 
 		if (!host->tuning_done) {

Searching a Bug on Raspberry Pi

[Georg Gast]

Hi there!

Currently i try to locate a bug in sdhci.c on RaspberryPi. I know it is 
not the mainline kernel...

Currently i use Kernel 3.8.13 and RT Patch 3.8.13-rt12 on the kernel 
found at git://github.com/raspberrypi/linux.git on branch rpi-3.8.y.

The bug shows up as booting and after 3 seconds of booting as it wants 
to mount the rootfs in a deadlock. as i compiled sdhci_bcm2708 as a 
module i can boot the kernel and system as i pushed the system to a usb 
stick. As far as good.

Now i try to find the bug, if the rootfs is on the sdcard. I narrowd the 
deadlock down to the following lines:

void sdhci_spin_lock_irqsave(struct sdhci_host *host,unsigned long *flags)
{
     return;
#ifdef CONFIG_PREEMPT
     if(enable_llm)
     {
         while(sdhci_locked)
         {
             preempt_schedule();
         }
         spin_lock_irqsave(&host->lock,*flags);
         disable_irq(host->irq);
         if(host->second_irq)
             disable_irq(host->second_irq);
         local_irq_enable();
     }
     else
#endif
         spin_lock_irqsave(&host->lock,*flags);
}

void sdhci_spin_unlock_irqrestore(struct sdhci_host *host,unsigned long 
flags)
{
     return;
#ifdef CONFIG_PREEMPT
     if(enable_llm)
     {
         local_irq_disable();
         if(host->second_irq)
             enable_irq(host->second_irq);
         enable_irq(host->irq);
     }
#endif
     spin_unlock_irqrestore(&host->lock,flags);
}

in drivers/mmc/host/sdhci.c . The both return statements are added by 
me. Now i can boot the RT Preempt kernel but it is for sure not the 
right fix.

My problem is, that i dont know how those both functions should work 
with the rt preempt patch.

What can you suggest me to fix that deadlock? I already bought the book 
"Linux Kernel Development" but i dont know how the kernel works exaktly 
with the PREEMPT patch. Can you point me to a site which shows how 
RTPREEMPT works in the irqsave/restore functions or can you please 
describe it to me?

Thank you
Georg

Re: Searching a Bug on Raspberry Pi

[Kirill Tkhai]

Hi, Georg!

29.06.2013, 22:39, "Georg Gast" <georg@schorsch-tech.de>:
> Hi there!
>
> Currently i try to locate a bug in sdhci.c on RaspberryPi. I know it is
> not the mainline kernel...
>
> Currently i use Kernel 3.8.13 and RT Patch 3.8.13-rt12 on the kernel
> found at git://github.com/raspberrypi/linux.git on branch rpi-3.8.y.
>
> The bug shows up as booting and after 3 seconds of booting as it wants
> to mount the rootfs in a deadlock. as i compiled sdhci_bcm2708 as a
> module i can boot the kernel and system as i pushed the system to a usb
> stick. As far as good.
>
> Now i try to find the bug, if the rootfs is on the sdcard. I narrowd the
> deadlock down to the following lines:
>
> void sdhci_spin_lock_irqsave(struct sdhci_host *host,unsigned long *flags)
> {
>      return;
> #ifdef CONFIG_PREEMPT
>      if(enable_llm)
>      {
>          while(sdhci_locked)
>          {
>              preempt_schedule();
>          }
>          spin_lock_irqsave(&host->lock,*flags);
>          disable_irq(host->irq);
>          if(host->second_irq)
>              disable_irq(host->second_irq);
>          local_irq_enable();
>      }
>      else
> #endif
>          spin_lock_irqsave(&host->lock,*flags);
> }
>
> void sdhci_spin_unlock_irqrestore(struct sdhci_host *host,unsigned long
> flags)
> {
>      return;
> #ifdef CONFIG_PREEMPT
>      if(enable_llm)
>      {
>          local_irq_disable();
>          if(host->second_irq)
>              enable_irq(host->second_irq);
>          enable_irq(host->irq);
>      }
> #endif
>      spin_unlock_irqrestore(&host->lock,flags);
> }
>
> in drivers/mmc/host/sdhci.c . The both return statements are added by
> me. Now i can boot the RT Preempt kernel but it is for sure not the
> right fix.
>
> My problem is, that i dont know how those both functions should work
> with the rt preempt patch.

Spinlocks transform to mutexes on RT kernel and really functions like
spin_lock_irqsave() do not disable interrupts. Your problem can be connected
with that.

If you code works on !RT kernel you should convert the type of host->lock
from spinlock_t to raw_spinlock_t and all of spin_lock_irqsave()
to raw_spin_lock_irqsave() etc.

I think this will help you. Sadly, I don't have enough time to look at whole of your
driver.

Kirill
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html