QSD8250 illegal instruction

QSD8250 illegal instruction

[lukas]

Hello,

I'm not sure if this is the right place but I don't know who else to  
address with this problem. The users of this list seem to know much  
about the qsd8* platform.

We are porting linux to the qsd8250 based htc leo device.

The code can be found at  
git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git  
htc-msm-2.6.32
web:  
http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=shortlog;h=refs/heads/htc-msm-2.6.32

It is based on the android-msm-2.6.32 branch from the google git at:  
http://android.git.kernel.org/?p=kernel/msm.git;a=shortlog;h=refs/heads/android-msm-2.6.32

General information on the device and on this subject can be found  
here: http://htc-linux.org/wiki/index.php?title=Leo


A lot of stuff is working out of the box. Our big problem is that we  
get illegal instructions on userspace binary execution. Some program  
work well, others will always crash with segfault / illegal  
instruction errors. For a while we thought that it is a question of  
using the correct toolchain and flags to compile. But it seems that  
the errors occur with various configurations. Also we tested  
rootfilesystems that worked perfectly well on the htc passion aka  
nexus one (same qsd8250) and they crashed in the same way.
This makes it look like we have a problem with our kernel or that  
there are some hardware differences.

My understanding about these low-level things is bad so I can't go  
into great detail. I would like to to take the liberty to refer to the  
mailing list where the problem is discussed:

http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/thread.html

It would be very nice if somebody could take a look at that. We are  
stI will paste the mail with the latest finding for easier reference.

On Fri, May 28, 2010 at 3:09 PM, Le Tama <letama at free.fr> wrote:

> Hi Guys,
>
> I did a new experiment that yielded interesting results.
>
> Here is what I did: I compiled eclair qaesd for ffa to get android binaries
> with android toolchain.
>
> With these binaries, I did a init script using android sh binary, it
> reaches
> the prompt.
>
> I also launched static adbd from this script, that was terminated because
> of
> invalid instruction:
>
> [   11.376047] adbd (42): undefined instruction: pc=0000fc60
> [   11.380015] Code: e5843000 e8bd8010 e59f102c e1a0c000 (e52d4004)
> [   11.387759] PL:send_signal (sig=4, from_ancestor=0)
>
>
> I dumped the compiled adbd with objdump, here is the location:
>
> 0000fc58 <pthread_attr_init>:
>    fc58:       e59f102c        ldr     r1, [pc, #44]   ; fc8c
> <pthread_attr_init+0x34>
>    fc5c:       e1a0c000        mov     ip, r0
>    fc60:       e52d4004        push    {r4}            ; (str r4, [sp,
> #-4]!)
>    fc64:       e59f4024        ldr     r4, [pc, #36]   ; fc90
> <pthread_attr_init+0x38>
>    fc68:       e08f3001        add     r3, pc, r1
>    fc6c:       e0834004        add     r4, r3, r4
>    fc70:       e8b4000f        ldm     r4!, {r0, r1, r2, r3}
>    fc74:       e8ac000f        stmia   ip!, {r0, r1, r2, r3}
>    fc78:       e8940003        ldm     r4, {r0, r1}
>    fc7c:       e88c0003        stm     ip, {r0, r1}
>    fc80:       e3a00000        mov     r0, #0
>    fc84:       e8bd0010        pop     {r4}
>    fc88:       e12fff1e        bx      lr
>    fc8c:       0001890c        .word   0x0001890c
>    fc90:       ffffbe04        .word   0xffffbe04
>
> So, as you can see, the undefined instruction is the push {r4} line.
>
> If I interpret well what I see, the instruction is legit, memory is ok, but
> the cpu considers it invalid.
>
> Now, the question is why ? Invalid instruction cache ? Wrong processor
> state
> ?
>
> This is out of my league, if someone has an explanation ?
>
> Best,
>
> LeTama
>
>
>
> _______________________________________________
> Mobile-linux-discuss mailing list
> Mobile-linux-discuss at linuxtogo.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/mobile-linux-discuss
>

We've been struggeling to get past this problem for quite a while now.  
Any hint is welcome!

Kind regards,

Lukas

QSD8250 illegal instruction

[lukas]

Hello,

I'm not sure if this is the right place but I don't know who else to  
address with this problem. The users of this list seem to know much  
about the qsd8* platform.

We are porting linux to the qsd8250 based htc leo device.

The code can be found at
git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git htc-msm-2.6.32
web:  
http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=shortlog;h=refs/heads/htc-msm-2.6.32

It is based on the android-msm-2.6.32 branch from the google git at:  
http://android.git.kernel.org/?p=kernel/msm.git;a=shortlog;h=refs/heads/android-msm-2.6.32

General information on the device and on this subject can be found  
here: http://htc-linux.org/wiki/index.php?title=Leo


Our big problem is that we get illegal instructions on userspace  
binary execution. Some program work well, others will always crash  
with segfault / illegal instruction errors. For a while we thought  
that it is a question of using the correct toolchain and flags to  
compile. But it seems that the errors occur with various  
configurations. Also we tested rootfilesystems that worked perfectly  
well on the htc passion aka nexus one (same qsd8250) and they crashed  
in the same way. This makes it look like we rather have a problem with  
our kernel or that there are some hardware differences.

My understanding about these low-level things is bad so I can't go  
into great detail. I would like to take the liberty to refer to the  
mailing list where the problem is discussed:

http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/thread.html

It would be very nice if somebody could take a look at that. I will  
paste the mail with the latest finding for easier reference.

On Fri, May 28, 2010 at 3:09 PM, Le Tama <letama at free.fr> wrote:

> Hi Guys,
>
> I did a new experiment that yielded interesting results.
>
> Here is what I did: I compiled eclair qaesd for ffa to get android binaries
> with android toolchain.
>
> With these binaries, I did a init script using android sh binary, it
> reaches
> the prompt.
>
> I also launched static adbd from this script, that was terminated because
> of
> invalid instruction:
>
> [   11.376047] adbd (42): undefined instruction: pc=3D0000fc60
> [   11.380015] Code: e5843000 e8bd8010 e59f102c e1a0c000 (e52d4004)
> [   11.387759] PL:send_signal (sig=3D4, from_ancestor=3D0)
>
>
> I dumped the compiled adbd with objdump, here is the location:
>
> 0000fc58 <pthread_attr_init>:
>    fc58:       e59f102c        ldr     r1, [pc, #44]   ; fc8c
> <pthread_attr_init+0x34>
>    fc5c:       e1a0c000        mov     ip, r0
>    fc60:       e52d4004        push    {r4}            ; (str r4, [sp,
> #-4]!)
>    fc64:       e59f4024        ldr     r4, [pc, #36]   ; fc90
> <pthread_attr_init+0x38>
>    fc68:       e08f3001        add     r3, pc, r1
>    fc6c:       e0834004        add     r4, r3, r4
>    fc70:       e8b4000f        ldm     r4!, {r0, r1, r2, r3}
>    fc74:       e8ac000f        stmia   ip!, {r0, r1, r2, r3}
>    fc78:       e8940003        ldm     r4, {r0, r1}
>    fc7c:       e88c0003        stm     ip, {r0, r1}
>    fc80:       e3a00000        mov     r0, #0
>    fc84:       e8bd0010        pop     {r4}
>    fc88:       e12fff1e        bx      lr
>    fc8c:       0001890c        .word   0x0001890c
>    fc90:       ffffbe04        .word   0xffffbe04
>
> So, as you can see, the undefined instruction is the push {r4} line.
>
> If I interpret well what I see, the instruction is legit, memory is ok, but
> the cpu considers it invalid.
>
> Now, the question is why ? Invalid instruction cache ? Wrong processor
> state
> ?
>
> This is out of my league, if someone has an explanation ?
>
> Best,
>
> LeTama
>
>
>
> _______________________________________________
> Mobile-linux-discuss mailing list
> Mobile-linux-discuss at linuxtogo.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/mobile-linux-discuss
>

We've been struggeling to get past this problem for quite a while now.  
Any hint is welcome.

Kind regards,

Lukas

Re: QSD8250 illegal instruction

[David Brown]

On Sat, May 29, 2010 at 08:20:33PM +0100, lukas@htc-linux.org wrote:

> I'm not sure if this is the right place but I don't know who else to  
> address with this problem. The users of this list seem to know much  
> about the qsd8* platform.

Yes, this is the correct list.  There should be several
knowledgeable people who can chime in here.

To start with, can you share the .config you built your kernel
with?

Thanks,
David

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: QSD8250 illegal instruction

[lukas]

Hello,

Quoting David Brown <davidb@codeaurora.org>:
>
> To start with, can you share the .config you built your kernel
> with?
>
> Thanks,
> David

We are using 'arch/arm/configs/htcleo_defconfig' from the  
htc-msm-2.6.32 branch in
git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git repository.

Here is a direct link
http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/configs/htcleo_defconfig;h=4d3141dae827c28a0cb46b04a3a075bf489cd842;hb=refs/heads/htc-msm-2.6.32

Thanks a lot,

Lukas

Re: QSD8250 illegal instruction

[Lukas-David Gorris]

Quoting lukas@htc-linux.org:

> Hello,
>
> Quoting David Brown <davidb@codeaurora.org>:
>>
>> To start with, can you share the .config you built your kernel
>> with?
>>
>> Thanks,
>> David
>
> We are using 'arch/arm/configs/htcleo_defconfig' from the
> htc-msm-2.6.32 branch in
> git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git repository.
>
> Here is a direct link
> http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/configs/htcleo_defconfig;h=4d3141dae827c28a0cb46b04a3a075bf489cd842;hb=refs/heads/htc-msm-2.6.32
>
> Thanks a lot,
>
> Lukas

Hello,

As I think this got under a bit due to the defconfig discussion I'm  
just bumping it up. Also LeTama seems to have confirmed the thing with  
the failing push instruction in a different case:

http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000095.html

I will just quote again for easier reference.


Le Tama letama at free.fr wrote on
Sun May 30 19:56:23 CEST 2010

Re All,

I did another experiment with the android binaries, this one native android
init with a cleaned init.rc.

This time, it results in segmentation fault, I added few traces to the
kernel:

[    9.520195] PL:kernel_execve(/init)
[    9.524717] PL:do_translation_fault
[    9.529754] PL:do_translation_fault
[    9.579853] init: cannot open '/initlogo.rle'
[    9.609789] lcdc_unblank: ()
[    9.676751] mdp irq already on 4000 4000
[    9.676762] mdp_dma: busy
[    9.725516] PL:do_translation_fault
[    9.734390] msm72k_udc: msm72k_udc: portchange USB_SPEED_HIGH
[    9.743216] msm72k_udc: msm72k_udc: reset
[    9.747724] PL:do_translation_fault
[    9.752509] PL:do_translation_fault
[    9.757389] PL:do_translation_fault
[    9.762623] PL:send_signal (sig=17, from_ancestor=0)
[    9.767116] PL: unlikely(vma->vm_start > addr)1
[    9.771485] PL:vma->vm_start = 8000
[    9.775757] PL:vma->vm_end   = 20000
[    9.780001] PL:vma->vm_flags = 8001875
[    9.784192] PL:addr          = 0
[    9.788386] PL:SIGSEGV/SEC_MAPERR
[    9.792554] pgd = c655c000
[    9.796654] [00000000] *pgd=26555031, *pte=00000000, *ppte=00000000
[    9.804887]
[    9.808952] Pid: 1, comm:                 init
[    9.813014] CPU: 0    Not tainted  (2.6.32.9-38056-gee54d02-dirty #146)
[    9.821087] PC is at 0x15a74
[    9.825101] LR is at 0xb2f9
[    9.829050] pc : [<00015a74>]    lr : [<0000b2f9>]    psr: 00000030
[    9.829060] sp : bea3bdd0  ip : ff0a0000  fp : 00000000
[    9.840921] r10: 00000000  r9 : 00000000  r8 : 00000000
[    9.844872] r7 : 0001c6fc  r6 : 0000068d  r5 : 00000006  r4 : 0002048c
[    9.852629] mdp irq already on 4000 4000
[    9.856538] mdp_dma: busy
[    9.860403] r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 0000068f
[    9.868101] Flags: nzcv  IRQs on  FIQs on  Mode USER_32  ISA Thumb
Segment user
[    9.875816] PL: instruction=4c9eb5f0
[    9.879668] Control: 10c5787d  Table: 2655c019  DAC: 00000015
[    9.887370]
[    9.887374] IP: 0xff09ff80:


The init disassembly gives me this:
0000b2c4 <read_file>: (the caller as shown by LR, in thumb)
		....
     b2f0:       b9f8            cbnz    r0, b332 <read_file+0x6e>
     b2f2:       1cb0            adds    r0, r6, #2
     b2f4:       f00a fbbe       bl      15a74 <malloc>
     b2f8:       4604            mov     r4, r0
             ....

00015a74 <malloc>: (crash location, just at the beginning, in thumb too)
    15a74:       b5f0            push    {r4, r5, r6, r7, lr}

    15a76:       4c9e            ldr     r4, [pc, #632]  ; (15cf0
<malloc+0x27c>)

    15a78:       a200            add     r2, pc, #0      ; (adr r2, 15a7c
<malloc+0x8>)

    15a7a:       499e            ldr     r1, [pc, #632]  ; (15cf4
<malloc+0x280>)

    15a7c:       18a4            adds    r4, r4, r2

    15a7e:       4605            mov     r5, r0

    15a80:       1860            adds    r0, r4, r1
                   .....
    15cea:       4630            mov     r0, r6

    15cec:       b007            add     sp, #28

    15cee:       bdf0            pop     {r4, r5, r6, r7, pc}

    15cf0:       0000aa10        .word   0x0000aa10

    15cf4:       000012ec        .word   0x000012ec



00015cf8 <calloc>:



I don't know if it's a coincidence, but again, a push instruction.


Here:
[    9.767116] PL: unlikely(vma->vm_start > addr)1
[    9.771485] PL:vma->vm_start = 8000
[    9.775757] PL:vma->vm_end   = 20000
[    9.780001] PL:vma->vm_flags = 8001875
[    9.784192] PL:addr          = 0

And here:
[    9.796654] [00000000] *pgd=26555031, *pte=00000000, *ppte=00000000

, it shows that the instruction at PC tries to access address 0

Why, I have no clue.

In memory, the code at 00015a74 is really the push instruction like
displayed here (this is *(PC)):
[    9.875816] PL: instruction=4c9eb5f0


Nothing really new here, but it could give hints for the arm guys ?

By the way, Dcordes, you asked them for help, right ? The thread can be seen
somewhere ?

Best,

LeTama



On a side note, we tried to use different snapdragon kernel  
repositories like some 2.6.29 at codeaurora and the problem seems to  
be the same.

Also might be interesting that we ran into some timer problem with the  
kernel configurations of other qsd8250 machines. Nexus one is using GP  
timer. If we set that on htcleo machine, system will freeze soon as  
there is some 'waiting code'. E.g. setting rootwait=n or rootdelay in  
cmdline always caused a panic with GP.
setting
'CONFIG_MSM7X00A_USE_DG_TIMER=y'
instead fixed that. No idea if that is any relevant but it might ring  
a bell for somebody who knows this SoC well.

Thank,

Lukas

Re: QSD8250 illegal instruction

[Lukas-David Gorris]

Hello,

Quoting lukas@htc-linux.org:

> Hello,
>
> Quoting David Brown <davidb@codeaurora.org>:
>>
>> To start with, can you share the .config you built your kernel
>> with?
>>
>> Thanks,
>> David
>
> We are using 'arch/arm/configs/htcleo_defconfig' from the
> htc-msm-2.6.32 branch in
> git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git repository.
>
> Here is a direct link
> http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/configs/htcleo_defconfig;h=4d3141dae827c28a0cb46b04a3a075bf489cd842;hb=refs/heads/htc-msm-2.6.32
>
> Thanks a lot,
>
> Lukas

As I think this got under a bit due to the ARM defconfig discussion  
I'm just bumping it up. Also LeTama seems to have confirmed the  
problem with the failing push instruction in a different case:

http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000095.html

I will just quote again for easier reference.


Le Tama letama at free.fr wrote on
Sun May 30 19:56:23 CEST 2010

Re All,

I did another experiment with the android binaries, this one native android
init with a cleaned init.rc.

This time, it results in segmentation fault, I added few traces to the
kernel:

[    9.520195] PL:kernel_execve(/init)
[    9.524717] PL:do_translation_fault
[    9.529754] PL:do_translation_fault
[    9.579853] init: cannot open '/initlogo.rle'
[    9.609789] lcdc_unblank: ()
[    9.676751] mdp irq already on 4000 4000
[    9.676762] mdp_dma: busy
[    9.725516] PL:do_translation_fault
[    9.734390] msm72k_udc: msm72k_udc: portchange USB_SPEED_HIGH
[    9.743216] msm72k_udc: msm72k_udc: reset
[    9.747724] PL:do_translation_fault
[    9.752509] PL:do_translation_fault
[    9.757389] PL:do_translation_fault
[    9.762623] PL:send_signal (sig=17, from_ancestor=0)
[    9.767116] PL: unlikely(vma->vm_start > addr)1
[    9.771485] PL:vma->vm_start = 8000
[    9.775757] PL:vma->vm_end   = 20000
[    9.780001] PL:vma->vm_flags = 8001875
[    9.784192] PL:addr          = 0
[    9.788386] PL:SIGSEGV/SEC_MAPERR
[    9.792554] pgd = c655c000
[    9.796654] [00000000] *pgd=26555031, *pte=00000000, *ppte=00000000
[    9.804887]
[    9.808952] Pid: 1, comm:                 init
[    9.813014] CPU: 0    Not tainted  (2.6.32.9-38056-gee54d02-dirty #146)
[    9.821087] PC is at 0x15a74
[    9.825101] LR is at 0xb2f9
[    9.829050] pc : [<00015a74>]    lr : [<0000b2f9>]    psr: 00000030
[    9.829060] sp : bea3bdd0  ip : ff0a0000  fp : 00000000
[    9.840921] r10: 00000000  r9 : 00000000  r8 : 00000000
[    9.844872] r7 : 0001c6fc  r6 : 0000068d  r5 : 00000006  r4 : 0002048c
[    9.852629] mdp irq already on 4000 4000
[    9.856538] mdp_dma: busy
[    9.860403] r3 : 00000000  r2 : 00000000  r1 : 00000000  r0 : 0000068f
[    9.868101] Flags: nzcv  IRQs on  FIQs on  Mode USER_32  ISA Thumb
Segment user
[    9.875816] PL: instruction=4c9eb5f0
[    9.879668] Control: 10c5787d  Table: 2655c019  DAC: 00000015
[    9.887370]
[    9.887374] IP: 0xff09ff80:


The init disassembly gives me this:
0000b2c4 <read_file>: (the caller as shown by LR, in thumb)
		....
     b2f0:       b9f8            cbnz    r0, b332 <read_file+0x6e>
     b2f2:       1cb0            adds    r0, r6, #2
     b2f4:       f00a fbbe       bl      15a74 <malloc>
     b2f8:       4604            mov     r4, r0
             ....

00015a74 <malloc>: (crash location, just at the beginning, in thumb too)
    15a74:       b5f0            push    {r4, r5, r6, r7, lr}

    15a76:       4c9e            ldr     r4, [pc, #632]  ; (15cf0
<malloc+0x27c>)

    15a78:       a200            add     r2, pc, #0      ; (adr r2, 15a7c
<malloc+0x8>)

    15a7a:       499e            ldr     r1, [pc, #632]  ; (15cf4
<malloc+0x280>)

    15a7c:       18a4            adds    r4, r4, r2

    15a7e:       4605            mov     r5, r0

    15a80:       1860            adds    r0, r4, r1
                   .....
    15cea:       4630            mov     r0, r6

    15cec:       b007            add     sp, #28

    15cee:       bdf0            pop     {r4, r5, r6, r7, pc}

    15cf0:       0000aa10        .word   0x0000aa10

    15cf4:       000012ec        .word   0x000012ec



00015cf8 <calloc>:



I don't know if it's a coincidence, but again, a push instruction.


Here:
[    9.767116] PL: unlikely(vma->vm_start > addr)1
[    9.771485] PL:vma->vm_start = 8000
[    9.775757] PL:vma->vm_end   = 20000
[    9.780001] PL:vma->vm_flags = 8001875
[    9.784192] PL:addr          = 0

And here:
[    9.796654] [00000000] *pgd=26555031, *pte=00000000, *ppte=00000000

, it shows that the instruction at PC tries to access address 0

Why, I have no clue.

In memory, the code at 00015a74 is really the push instruction like
displayed here (this is *(PC)):
[    9.875816] PL: instruction=4c9eb5f0


Nothing really new here, but it could give hints for the arm guys ?

By the way, Dcordes, you asked them for help, right ? The thread can be seen
somewhere ?

Best,

LeTama



On a side note, we tried to use different snapdragon kernel  
repositories like some 2.6.29 at codeaurora and the problem seems to  
be the same.

Also might be interesting that we ran into some timer problem with the  
kernel configurations of other qsd8250 machines. Nexus one is using GP  
timer. If we set that on htcleo machine, system will freeze soon as  
there is some 'waiting code'. E.g. setting rootwait=n or rootdelay in  
cmdline always caused a panic with GP.
setting
'CONFIG_MSM7X00A_USE_DG_TIMER=y'
instead fixed that. No idea if that is any relevant but it might ring  
a bell for somebody who knows this SoC well.

Thank,

Lukas

Re: QSD8250 illegal instruction

[Daniel Walker]

On Sun, 2010-06-06 at 03:53 +0100, Lukas-David Gorris wrote:
> Hello,
> 
> Quoting lukas@htc-linux.org:
> 
> > Hello,
> >
> > Quoting David Brown <davidb@codeaurora.org>:
> >>
> >> To start with, can you share the .config you built your kernel
> >> with?
> >>
> >> Thanks,
> >> David
> >
> > We are using 'arch/arm/configs/htcleo_defconfig' from the
> > htc-msm-2.6.32 branch in
> > git://git.linuxtogo.org/home/groups/mobile-linux/kernel.git repository.
> >
> > Here is a direct link
> > http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/configs/htcleo_defconfig;h=4d3141dae827c28a0cb46b04a3a075bf489cd842;hb=refs/heads/htc-msm-2.6.32

You have a CONFIG_MACH_HTCLEO option enabled, but i'm pretty sure that's
not in the android tree.. Have you added any code along with that
option?

Daniel

RE: QSD8250 illegal instruction

[Le Tama]

Hi Daniel,

Sorry to jump in, but as Lukas isn't around:

> You have a CONFIG_MACH_HTCLEO option enabled, but i'm pretty sure 
> that's not in the android tree.. Have you added any code along 
> with that option?

Yes, everything is here:

http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=tree;f=arch/arm/mach-msm;h=7574224b5ce1b78ede435eccd2037a90a385ceb4;hb=refs/heads/htc-msm-2.6.32


board-htcleo.c:

http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/mach-msm/board-htcleo.c;h=a5a8b976837c2b32338b55ea2f4d017c64133c78;hb=refs/heads/htc-msm-2.6.32

board-htcleo.h:

http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/mach-msm/board-htcleo.h;h=ece396d1c427004f5d778585e119115f12e4c393;hb=refs/heads/htc-msm-2.6.32

and few files along them, board-htcleo*, plus specific clocks in clock-wince.c

We have the same issue with a much less modified tree based on another kernel here:

http://gitorious.org/linux-on-htc-qsd/linux-on-htc-qsd/trees/master/arch/arm/mach-msm


Thanks,

LeTama

RE: QSD8250 illegal instruction

[Daniel Walker]

On Wed, 2010-06-09 at 22:18 +0200, Le Tama wrote:
> Hi Daniel,
> 
> Sorry to jump in, but as Lukas isn't around:
> 
> > You have a CONFIG_MACH_HTCLEO option enabled, but i'm pretty sure 
> > that's not in the android tree.. Have you added any code along 
> > with that option?
> 
> Yes, everything is here:
> 
> http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=tree;f=arch/arm/mach-msm;h=7574224b5ce1b78ede435eccd2037a90a385ceb4;hb=refs/heads/htc-msm-2.6.32
> 
> 
> board-htcleo.c:
> 
> http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/mach-msm/board-htcleo.c;h=a5a8b976837c2b32338b55ea2f4d017c64133c78;hb=refs/heads/htc-msm-2.6.32
> 
> board-htcleo.h:
> 
> http://git.linuxtogo.org/?p=groups/mobile-linux/kernel.git;a=blob;f=arch/arm/mach-msm/board-htcleo.h;h=ece396d1c427004f5d778585e119115f12e4c393;hb=refs/heads/htc-msm-2.6.32
> 
> and few files along them, board-htcleo*, plus specific clocks in clock-wince.c
> 
> We have the same issue with a much less modified tree based on another kernel here:
> 
> http://gitorious.org/linux-on-htc-qsd/linux-on-htc-qsd/trees/master/arch/arm/mach-msm

Have you tried reproducing this with a smaller userspace, like not the
full Android environment?

Daniel

RE: QSD8250 illegal instruction

[Lukas-David Gorris]

Hello,

Quoting Daniel Walker <dwalker@codeaurora.org>:

>
> Have you tried reproducing this with a smaller userspace, like not the
> full Android environment?
>
> Daniel
>

Yes. There were some approaches to produce userspace for the htcleo  
machine using openembedded buildsystem. Parts of the resulting  
binaries work, some fail with segfault / illegal instruction. I  
compiled most of this in the projects' bugtracking system which can be  
found here:
http://bugs.openembedded.org/show_bug.cgi?id=5435

Also LeTama produced a minimal busybox image. In a certain  
configuration it works.  
http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000087.html

I can't detail on this as I was not able to reproduce the working busybox.

Thanks for replying, LeTama.


Best regards

Lukas

RE: QSD8250 illegal instruction

[Daniel Walker]

On Thu, 2010-06-10 at 18:37 +0100, Lukas-David Gorris wrote:
> Hello,
> 
> Quoting Daniel Walker <dwalker@codeaurora.org>:
> 
> >
> > Have you tried reproducing this with a smaller userspace, like not the
> > full Android environment?
> >
> > Daniel
> >
> 
> Yes. There were some approaches to produce userspace for the htcleo  
> machine using openembedded buildsystem. Parts of the resulting  
> binaries work, some fail with segfault / illegal instruction. I  
> compiled most of this in the projects' bugtracking system which can be  
> found here:
> http://bugs.openembedded.org/show_bug.cgi?id=5435
> 
> Also LeTama produced a minimal busybox image. In a certain  
> configuration it works.  
> http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000087.html
> 
> I can't detail on this as I was not able to reproduce the working busybox.

Ok, well another thing you could try is disabling some of the drivers
and re-testing. One specifically is the framebuffer.

At this point since no one else has jumped up and said they know exactly
what this is, then you'll have to narrow it down a bit more.

Daniel

RE: QSD8250 illegal instruction

[Le Tama]

Hi Daniel,

> Also LeTama produced a minimal busybox image. In a certain  
> configuration it works.  
> http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000087.html

On my side, here is what I did, all these tests with initrd only to avoid sdcard driver interaction:

1) a busybox build with re-compiled executables with Codesourcery 2010q1 toolchain
- Statically linked executables were crashing, 
- Dynamically linked with arm v4 stdlib dlls, executables were apparently working. I'm not sure it was luck or a pattern, I didn't test a lots of executables in this configuration.
- Dynamically linked with thumb2 stdlib dlls, executables were crashing

At this point, I began suspecting toolchain issue, I tried multiple ones including android pre-built and custom made without success.

2) A simple C init executable that forks /execs a simple Write("helloworld") program.
	- Again, crashes, with multiple toolchains trials.

3) As I thought that maybe my toolchains were not fine with glibc (crashes often in unwind code) , I rebuilt android for QSD8250_FFA, trimmed down init.rc to launch only adbd and debuggerd from the standard android init. This is the one that is producing the reports you saw in the emails. It has been verified successfully on a nexus one with the same kernel, modified to revert htcleo specific clocks and board.

Best,

LeTama

RE: QSD8250 illegal instruction

[Daniel Walker]

On Thu, 2010-06-10 at 20:24 +0200, Le Tama wrote:
> Hi Daniel,
> 
> > Also LeTama produced a minimal busybox image. In a certain  
> > configuration it works.  
> > http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000087.html
> 
> On my side, here is what I did, all these tests with initrd only to
> avoid sdcard driver interaction:
> 
> 1) a busybox build with re-compiled executables with Codesourcery
> 2010q1 toolchain
> - Statically linked executables were crashing, 
> - Dynamically linked with arm v4 stdlib dlls, executables were
> apparently working. I'm not sure it was luck or a pattern, I didn't
> test a lots of executables in this configuration.
> - Dynamically linked with thumb2 stdlib dlls, executables were
> crashing
> 
> At this point, I began suspecting toolchain issue, I tried multiple
> ones including android pre-built and custom made without success.
> 
> 2) A simple C init executable that forks /execs a simple
> Write("helloworld") program.
> 	- Again, crashes, with multiple toolchains trials.
> 
> 3) As I thought that maybe my toolchains were not fine with glibc
> (crashes often in unwind code) , I rebuilt android for QSD8250_FFA,
> trimmed down init.rc to launch only adbd and debuggerd from the
> standard android init. This is the one that is producing the reports
> you saw in the emails. It has been verified successfully on a nexus
> one with the same kernel, modified to revert htcleo specific clocks
> and board.

What kind of kernel did you use? You say you wanted to "avoid sdcard
driver interaction" , but was the sdcard actually compiled out?

Daniel

RE: QSD8250 illegal instruction

[Daniel Walker]

On Thu, 2010-06-10 at 20:24 +0200, Le Tama wrote:
> Hi Daniel,
> 
> > Also LeTama produced a minimal busybox image. In a certain  
> > configuration it works.  
> > http://lists.linuxtogo.org/pipermail/mobile-linux-discuss/2010-May/000087.html
> 
> On my side, here is what I did, all these tests with initrd only to avoid sdcard driver interaction:
> 
> 1) a busybox build with re-compiled executables with Codesourcery 2010q1 toolchain
> - Statically linked executables were crashing, 
> - Dynamically linked with arm v4 stdlib dlls, executables were apparently working. I'm not sure it was luck or a pattern, I didn't test a lots of executables in this configuration.
> - Dynamically linked with thumb2 stdlib dlls, executables were crashing
> 
> At this point, I began suspecting toolchain issue, I tried multiple ones including android pre-built and custom made without success.
> 
> 2) A simple C init executable that forks /execs a simple Write("helloworld") program.
> 	- Again, crashes, with multiple toolchains trials.
> 
> 3) As I thought that maybe my toolchains were not fine with glibc (crashes often in unwind code) , I rebuilt android for QSD8250_FFA, trimmed down init.rc to launch only adbd and debuggerd from the standard android init. This is the one that is producing the reports you saw in the emails. It has been verified successfully on a nexus one with the same kernel, modified to revert htcleo specific clocks and board.

btw, we have added very very basic QSD8x50 support into 2.6.35-rc1 .. So
your welcome to port Leo support (or nexus one for that matter) onto the
mainline kernel and I can push that into 2.6.36 .. It would have to be
fairly basic support tho just booting with initrd and a uart.

Daniel

RE: QSD8250 illegal instruction

[Le Tama]

>What kind of kernel did you use? You say you wanted to "avoid sdcard
>driver interaction" , but was the sdcard actually compiled out?

No, the driver was not compiled out on this one, however I think the other kernel doesn't have it.

I will try removing the framebuffer from this one as it's almost unmodified and with almost no driver, but I think I tried that already.

LeTama

RE: QSD8250 illegal instruction

[Lukas-David Gorris]

Quoting Daniel Walker <dwalker@codeaurora.org>:

>
> What kind of kernel did you use? You say you wanted to "avoid sdcard
> driver interaction" , but was the sdcard actually compiled out?
>
> Daniel
>

In reply to this and to your general proposition to narrow down the  
problem I would like to mention that something like that was tried.  
Markinus who wrote many of the patches in the mentioned htc-msm-2.6.32  
branch made an approach to narrow down the problem by disabling  
everything possible in the kernel configuration and manually removing  
things from init, also SD. He still got the segfaults with a very very  
basic kernel Image.

Did you see my mention of the DB vs BG timer config problem? As we  
have to put a different config there than nexus, maybe there are some  
minor differences in the SoC. Maybe that could be related to the  
problem at hand.

With regard to that we had some discussion on QSD8250 vs QSD8250B .  
Maybe somebody can shed some light on this. Is there actually any  
difference in B vs non-B ? Is there an A ? Or is this something like  
MSM7200A vs msm7201A ?

Quoting Daniel Walker <dwalker@codeaurora.org>:

> btw, we have added very very basic QSD8x50 support into 2.6.35-rc1 .. So
> your welcome to port Leo support (or nexus one for that matter) onto the
> mainline kernel and I can push that into 2.6.36 .. It would have to be
> fairly basic support tho just booting with initrd and a uart.
>
> Daniel

I have been watching the QSD mainlining with excitement. If we ever  
get the problem solved, it would be very nice to add Leo in mainline.  
In that case I would be ready to maintain it and run frequent test  
builds etc.

Lukas

RE: QSD8250 illegal instruction

[Daniel Walker]

On Thu, 2010-06-10 at 20:33 +0100, Lukas-David Gorris wrote:
> Quoting Daniel Walker <dwalker@codeaurora.org>:
> 
> >
> > What kind of kernel did you use? You say you wanted to "avoid sdcard
> > driver interaction" , but was the sdcard actually compiled out?
> >
> > Daniel
> >
> 
> In reply to this and to your general proposition to narrow down the  
> problem I would like to mention that something like that was tried.  
> Markinus who wrote many of the patches in the mentioned htc-msm-2.6.32  
> branch made an approach to narrow down the problem by disabling  
> everything possible in the kernel configuration and manually removing  
> things from init, also SD. He still got the segfaults with a very very  
> basic kernel Image.

Ok .

> Did you see my mention of the DB vs BG timer config problem? As we  
> have to put a different config there than nexus, maybe there are some  
> minor differences in the SoC. Maybe that could be related to the  
> problem at hand.

I did see that.

> With regard to that we had some discussion on QSD8250 vs QSD8250B .  
> Maybe somebody can shed some light on this. Is there actually any  
> difference in B vs non-B ? Is there an A ? Or is this something like  
> MSM7200A vs msm7201A ?

I don't know of a QSD8250B SoC .. If there is one, I have no idea what
would be different.

It might be that the baseband processor is running different code
between the Nexus One and the Leo (i.e. different modem images), and
that code is programming the SoC differently between the two. Or it
could be a bootloader thing, does the Leo have fastboot ?

Daniel

RE: QSD8250 illegal instruction

[Lukas-David Gorris]

Quoting Daniel Walker <dwalker@codeaurora.org>:

> I don't know of a QSD8250B SoC .. If there is one, I have no idea what
> would be different.

Ok fine. So SoC revision seems like nothing to worry about.

> It might be that the baseband processor is running different code
> between the Nexus One and the Leo (i.e. different modem images), and
> that code is programming the SoC differently between the two.

Yes, we do have a different modem image. The Leo AMSS version is for  
WinCE. The device comes with windows mobile 6.5 installed.
Can you think of anything specifically regarding different SoC  
programming between different AMSS version? Some magic proc_comm to  
send?

> Or it could be a bootloader thing, does the Leo have fastboot ?

We are using http://htc-linux.org/wiki/index.php?title=HaRET to boot  
linux on Leo. So at boot time, all the hardware is initialized already  
by WinCE.

Lukas

RE: QSD8250 illegal instruction

[Daniel Walker]

On Thu, 2010-06-10 at 22:52 +0100, Lukas-David Gorris wrote:
> Quoting Daniel Walker <dwalker@codeaurora.org>:
> 
> > I don't know of a QSD8250B SoC .. If there is one, I have no idea what
> > would be different.
> 
> Ok fine. So SoC revision seems like nothing to worry about.

Where are you getting the B distinction from ? something your getting
out of WinCE ?

> > It might be that the baseband processor is running different code
> > between the Nexus One and the Leo (i.e. different modem images), and
> > that code is programming the SoC differently between the two.
> 
> Yes, we do have a different modem image. The Leo AMSS version is for  
> WinCE. The device comes with windows mobile 6.5 installed.
> Can you think of anything specifically regarding different SoC  
> programming between different AMSS version? Some magic proc_comm to  
> send?
> 
> > Or it could be a bootloader thing, does the Leo have fastboot ?
> 
> We are using http://htc-linux.org/wiki/index.php?title=HaRET to boot  
> linux on Leo. So at boot time, all the hardware is initialized already  
> by WinCE.

Ok .. I suppose there could be something different between what WinCE
expects, and what Linux expect. Do you know if this method has been used
successfully on other HTC phones with Qualcomm SoC's ?

Daniel

RE: QSD8250 illegal instruction

[Lukas-David Gorris]

Quoting Daniel Walker <dwalker@codeaurora.org>:

> On Thu, 2010-06-10 at 22:52 +0100, Lukas-David Gorris wrote:
>> Quoting Daniel Walker <dwalker@codeaurora.org>:
>>
>> > I don't know of a QSD8250B SoC .. If there is one, I have no idea what
>> > would be different.
>>
>> Ok fine. So SoC revision seems like nothing to worry about.
>
> Where are you getting the B distinction from ? something your getting
> out of WinCE ?

Yes, in the WinCE hardware information program it says CPU:  
QUALCOMM(R) QSD8250B .

>> > It might be that the baseband processor is running different code
>> > between the Nexus One and the Leo (i.e. different modem images), and
>> > that code is programming the SoC differently between the two.
>>
>> Yes, we do have a different modem image. The Leo AMSS version is for
>> WinCE. The device comes with windows mobile 6.5 installed.
>> Can you think of anything specifically regarding different SoC
>> programming between different AMSS version? Some magic proc_comm to
>> send?
>>
>> > Or it could be a bootloader thing, does the Leo have fastboot ?
>>
>> We are using http://htc-linux.org/wiki/index.php?title=HaRET to boot
>> linux on Leo. So at boot time, all the hardware is initialized already
>> by WinCE.
>
> Ok .. I suppose there could be something different between what WinCE
> expects, and what Linux expect. Do you know if this method has been used
> successfully on other HTC phones with Qualcomm SoC's ?
>

It has never been a problem to run linux using HaRET on msm7xxx  
(non-A) and msm7xxxA based WinCE devices. I don't know of any haret  
code for msm7* that handles low level initialization stuff other than  
things about starting the kernel (which we obviously do successfully  
on Leo). I will investigate this further.
But I can say I have never seen such a problem before. Once kernel on  
msm7* is started, things just work. No strange low level problems as  
seen on Leo I know of.

As for qsd8xxxx Leo is the first device we are booting linux on using haret.


Thanks

Lukas