Kernel boot problem on IXP422 Rev. A

Kernel boot problem on IXP422 Rev. A

[Marcus Tangermann]

Hello,

we currently try to boot a 2.6.21 kernel on a custom IXP422 based board. The boot loader (U-Boot) works fine so far, memory and flash test run successfully, the NPE is also able to ping a host. So we assume the board itself is more or less working at least.
When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow. We've figured out so far that the kernel dies somewhere between  the gunzip and start_kernel.
I would be glad for any hint to be able to solve the problem. 

Thank you a lot in advance.
Best regards
Marcus
_____________________________________________________________________
Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
http://smartsurfer.web.de/?mc=100071&distributionid=000000000066

Re: Kernel boot problem on IXP422 Rev. A

[Mike Frysinger]

On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
> we currently try to boot a 2.6.21 kernel

time to upgrade

> on a custom IXP422 based board.

you'll have better luck asking on the arm kernel mailing lists.  they
process this stuff every day.
-mike

Re: Kernel boot problem on IXP422 Rev. A

[Glenn Henshaw]

On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:

> On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
>> we currently try to boot a 2.6.21 kernel
>
> time to upgrade

   Wrong answer!!!

   Many embedded devices can't upgrade kernels easily because of  
customer requirements and certifications. For example, I have worked  
on Linux based applications in the financial industry. A kernel  
upgrade here is viewed as equivalent to switching from Windows XP to  
Vista, and requires significant effort in certification testing from  
the customer's perspective. This doesn't make economic sense for  
either party.

   Often developers not working in the embedded space don't understand  
this reality.

   Helpful responses on what does/does not work would be more  
appreciated.

    ... Glenn

>
>
>> on a custom IXP422 based board.
>
> you'll have better luck asking on the arm kernel mailing lists.  they
> process this stuff every day.
> -mike
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Glenn Henshaw                     Logical Outcome Ltd.
e: thraxisp@logicaloutcome.ca     w: www.logicaloutcome.ca

Re: Kernel boot problem on IXP422 Rev. A

[Mike Frysinger]

On Thu, Jun 12, 2008 at 5:28 PM, Glenn Henshaw wrote:
> On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:
>> On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
>>> we currently try to boot a 2.6.21 kernel
>>
>> time to upgrade
>
>  Wrong answer!!!

not really

>  Many embedded devices can't upgrade kernels easily because of customer
> <snip>

and there are plenty of *wrong* reasons that people are using old kernels.

>  Often developers not working in the embedded space don't understand this
> reality.

if i'm not classified as an embedded developer, then i dont know what one is :P

>  Helpful responses on what does/does not work would be more appreciated.

i gave a realistic one and i sent him to the right place -- the arm
kernel mailing lists
-mike

Re: Kernel boot problem on IXP422 Rev. A

[Craig Hollabaugh]

Glenn,
I totally agree with your certification statement statement as a legit
reason to not upgrade. I once worked for a aircraft lighting supplier,
their FAA certification testing process took about 4 years. Going
through the cert process makes one more resistant to change.
Craig




On Thu, 2008-06-12 at 17:28 -0400, Glenn Henshaw wrote:
> On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:
> 
> > On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
> >> we currently try to boot a 2.6.21 kernel
> >
> > time to upgrade
> 
>    Wrong answer!!!
> 
>    Many embedded devices can't upgrade kernels easily because of  
> customer requirements and certifications. For example, I have worked  
> on Linux based applications in the financial industry. A kernel  
> upgrade here is viewed as equivalent to switching from Windows XP to  
> Vista, and requires significant effort in certification testing from  
> the customer's perspective. This doesn't make economic sense for  
> either party.
> 
>    Often developers not working in the embedded space don't understand  
> this reality.
> 
>    Helpful responses on what does/does not work would be more  
> appreciated.
> 
>     ... Glenn
> 
> >
> >
> >> on a custom IXP422 based board.
> >
> > you'll have better luck asking on the arm kernel mailing lists.  they
> > process this stuff every day.
> > -mike
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux- 
> > embedded" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
-- 
Dr. Craig Hollabaugh, craig@hollabaugh.com, 970 240 0509
Author of Embedded Linux: Hardware, Software and Interfacing
www.embeddedlinuxinterfacing.com

Re: Kernel boot problem on IXP422 Rev. A

[Alexey Zaytsev]

On Fri, Jun 13, 2008 at 1:28 AM, Glenn Henshaw
<thraxisp@logicaloutcome.ca> wrote:
>
> On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:
>
>> On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
>>>
>>> we currently try to boot a 2.6.21 kernel
>>
>> time to upgrade
>
>  Wrong answer!!!

While the answer may be not valid any time you see
somebody complaining about an relatively old kernel not
working, it looks perfectly valid here.

They are experimenting with a new board, (well, not all that
new, probably), and a year-old kernel does not work. The
obvous thing it to try the latest kernel and see it it's working.
Should at least give a clue about what's wrong.

Re: Kernel boot problem on IXP422 Rev. A

[David Woodhouse]

On Thu, 2008-06-12 at 17:28 -0400, Glenn Henshaw wrote:
> > On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
> >> we currently try to boot a 2.6.21 kernel
> >
> > time to upgrade
> 
>    Wrong answer!!!
> 
>    Many embedded devices can't upgrade kernels easily because of  
> customer requirements and certifications. For example, I have worked  
> on Linux based applications in the financial industry. A kernel  
> upgrade here is viewed as equivalent to switching from Windows XP to  
> Vista, and requires significant effort in certification testing from  
> the customer's perspective. This doesn't make economic sense for  
> either party.

If this certification was granted despite Marcus' admission that the
kernel doesn't even boot -- it dies between gunzip and start_kernel --
then I suspect it wasn't the kind of certification which takes 4 years
to achieve.

_Most_ people who are having trouble with old kernels have extremely
_bad_ reasons for not updating.

-- 
dwmw2

Re: Kernel boot problem on IXP422 Rev. A

[Glenn Henshaw]

On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:

> On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
>> we currently try to boot a 2.6.21 kernel
>> on a custom IXP422 based board.
>

   What bootloader are you using? It may be that the memory controller  
is initialized improperly.

   For other resources, try the NSLU2 groups/pages. It runs the same  
processor and kernel (actually many 2.6 variants).

     ... Glenn
-- 
Glenn Henshaw                     Logical Outcome Ltd.
e: thraxisp@logicaloutcome.ca     w: www.logicaloutcome.ca

Re: Kernel boot problem on IXP422 Rev. A

[Greg Ungerer]

Hi Marcus,

Marcus Tangermann wrote:
> we currently try to boot a 2.6.21 kernel on a custom IXP422 based board. The boot loader (U-Boot) works fine so far, memory and flash test run successfully, the NPE is also able to ping a host. So we assume the board itself is more or less working at least.
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow. We've figured out so far that the kernel dies somewhere between  the gunzip and start_kernel.
> I would be glad for any hint to be able to solve the problem. 

The most common problem like this I have seen are that the
ARM machine type set by the boot loader does not match those
compiled into the kernel. Also make sure that the RAM
size being passed to the kernel is correct too.

Enable the kernels CONFIG_DEBUG_LL and it may give you some
early debug output (like an id not supported message). That
assumes a serial console...

Regards
Greg



------------------------------------------------------------------------
Greg Ungerer  --  Chief Software Dude       EMAIL:     gerg@snapgear.com
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Re: Kernel boot problem on IXP422 Rev. A

[George G. Davis]

On Thu, Jun 12, 2008 at 09:35:16PM +0200, Marcus Tangermann wrote:
> Hello,
> 
> we currently try to boot a 2.6.21 kernel on a custom IXP422 based board. The boot loader (U-Boot) works fine so far, memory and flash test run successfully, the NPE is also able to ping a host. So we assume the board itself is more or less working at least.
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow. We've figured out so far that the kernel dies somewhere between  the gunzip and start_kernel.
> I would be glad for any hint to be able to solve the problem. 

Try enabling DEBUG_LL to see if it's an machine ID error.  If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target
or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call
printascii() (enabled for the DEBUG_LL case) to print directly to the
serial port w/o a driver, etc.,.  You can find more details on these
low-level debugging hacks via a little googling...

HTH!

--
Regards,
George
> 
> Thank you a lot in advance.
> Best regards
> Marcus
> _____________________________________________________________________
> Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
> http://smartsurfer.web.de/?mc=100071&distributionid=000000000066
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel boot problem on IXP422 Rev. A

[Rob Landley]

On Thursday 12 June 2008 16:28:48 Glenn Henshaw wrote:
> On 12-Jun-08, at 4:04 PM, Mike Frysinger wrote:
> > On Thu, Jun 12, 2008 at 3:35 PM, Marcus Tangermann wrote:
> >> we currently try to boot a 2.6.21 kernel
> >
> > time to upgrade
>
>    Wrong answer!!!

Fairly common answer actually.  It translates to:

There's a decent chance whatever bug you're encountering is one we've already 
fixed.  As volunteers, we're not really interested in reinventing the wheel.  
We're trying to come up with a better kernel, and debugging old version may 
help you but it doesn't help us.

If you try the current version and the problem persists, we're happy to help 
you debug it, because this makes the kernel better for all of us.  If you 
have your own reason to want to stick with an old version, there are plenty 
of vendors happy to take your money to make this work for you, but you've 
sacrificed the support of the volunteer community in doing so.

>    Many embedded devices can't upgrade kernels easily because of
> customer requirements and certifications.

I'm aware of this.  That's what vendors are for.

> For example, I have worked 
> on Linux based applications in the financial industry. A kernel
> upgrade here is viewed as equivalent to switching from Windows XP to
> Vista, and requires significant effort in certification testing from
> the customer's perspective. This doesn't make economic sense for
> either party.

Making economic sense and making sense to open source volunteers are two 
different sets of criteria.  Part of "making economic sense" is realizing 
that you must replace volunteer effort with paid effort to do it that way.

The procedure is:

1) Try to reproduce the bug under a current kernel.  (Set up a _test_ system.)

If you can't reproduce it:
  Git bisect to find the fix, then backport it to your kernel if possible.
If you can reproduce it:
  Demonstrate the bug to the community in the current kernel, get help 
debugging the problem, then backport the fix to your kernel if possible.

>    Often developers not working in the embedded space don't understand
> this reality.

Often developers who _are_ working in the embedded space don't understand how 
and why volunteers collaborate through the internet in this thing 
called "open source development".

>    Helpful responses on what does/does not work would be more
> appreciated.

He did give a helpful response.  You didn't understand it.  I elaborated.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Kernel boot problem on IXP422 Rev. A

[Marcus Tangermann]

Hi Greg,

first, thanks for the hints.
>
> The most common problem like this I have seen are that the
> ARM machine type set by the boot loader does not match those
> compiled into the kernel. Also make sure that the RAM
> size being passed to the kernel is correct too.
>
This seems to be set correctly.

> Enable the kernels CONFIG_DEBUG_LL and it may give you some
> early debug output (like an id not supported message). That
> assumes a serial console...
Will try that,
Today I downloaded the snapgear distribution and tried whether i can 
generate a kernel image that works with it...and it boots (2.4 and 2.6)  
:-) Then I downloaded a fresh 2.6.25 kernel and compiled it with 
toolchain from the snapgear homepage...and it does not boot. Are there 
any special patches in the 2.6.19-uc1 from the snapgear distribution 
that might have influence on a IXP422 based board?
Additionally, it seems that we may have a little problem in the 
bootloader (uboot) when passing kernel parameters (e.g."setenv bootargs 
console=ttyS0,115200"). Setting such a paramter causes the same problem 
with a dying kernel. Maybe there is a relation.

Regards
Marcus

Re: Kernel boot problem on IXP422 Rev. A

[Tim Bird]

Rob,

This is an excellent and concise description of the open
source perspective on the problem.  I'll add just one note below.

Rob Landley wrote:
> 1) Try to reproduce the bug under a current kernel.  (Set up a _test_ system.)

This sounds easy, but can be quite difficult.

Very often, product developers are several versions behind, with
no easy way to use the current kernel version.  For example, a
common scenario is starting with a kernel that comes with a board
(with source mind you), where the kernel came from the semi-conductor
vendor, who paid a Linux vendor to do a port, and it was
released in a time-frame relative to the Linux vendor's
product schedule.

This is how you end up having people STARTING projects today
using a 2.6.11 kernel.  (I know of many).

The real difficulty, when a developer finds themselves in
this position, is how to forward-port the BSP code necessary to
reproduce the bug in the current kernel.  Often, the code
is not isolated well enough (this is a vendor problem that
really needs attention.  If you have the BSP in patches, it
is usually not too bad to forward port even across several
kernel versions.  But many vendors don't ship stuff this way.)

The fact is, that by a series of small steps and delays by
the linux vendor, chip vendor, board vendor,
and product developer the code is out-of step.

It's easy to say "don't get in this position", but
this even happens when everyone is playing nice and actively
trying to mainline stuff.  BSP support in arch trees often
lag mainline by a version or two.

The number of parties involved here is why, IMHO, it has
taken so long to make improvements in this area.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Kernel boot problem on IXP422 Rev. A

[Greg Ungerer]

Hi Marcus,

Marcus Tangermann wrote:
> first, thanks for the hints.
>>
>> The most common problem like this I have seen are that the
>> ARM machine type set by the boot loader does not match those
>> compiled into the kernel. Also make sure that the RAM
>> size being passed to the kernel is correct too.
>>
> This seems to be set correctly.
> 
>> Enable the kernels CONFIG_DEBUG_LL and it may give you some
>> early debug output (like an id not supported message). That
>> assumes a serial console...
> Will try that,
> Today I downloaded the snapgear distribution and tried whether i can 
> generate a kernel image that works with it...and it boots (2.4 and 2.6)  
> :-) Then I downloaded a fresh 2.6.25 kernel and compiled it with 
> toolchain from the snapgear homepage...and it does not boot. Are there 
> any special patches in the 2.6.19-uc1 from the snapgear distribution
> that might have influence on a IXP422 based board?

I don't recall any specific fixes as such. There is support
for a few extra IXPxxx based boards.

Do a quick diff against a stock 2.6.19. There may be a number of
changes related to non-mmu arm in there, they won't have any impact
on IXP4xx support though.


> Additionally, it seems that we may have a little problem in the 
> bootloader (uboot) when passing kernel parameters (e.g."setenv bootargs 
> console=ttyS0,115200"). Setting such a paramter causes the same problem 
> with a dying kernel. Maybe there is a relation.

Kernel dying, not just no console?
Is your serial console on ttyS0 or ttyS1?

Regards
Greg



------------------------------------------------------------------------
Greg Ungerer  --  Chief Software Dude       EMAIL:     gerg@snapgear.com
SnapGear -- a Secure Computing Company      PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com

Re: Kernel boot problem on IXP422 Rev. A

[Rob Landley]

On Friday 13 June 2008 15:05:54 Tim Bird wrote:
> Rob,
>
> This is an excellent and concise description of the open
> source perspective on the problem.  I'll add just one note below.
>
> Rob Landley wrote:
> > 1) Try to reproduce the bug under a current kernel.  (Set up a _test_
> > system.)
>
> This sounds easy, but can be quite difficult.

It's not a question of difficult or easy: it's the procedure that works.

You don't get support from a commercial vendor unless you pay them money, and 
you don't get support from open source developers unless you help us make the 
next release just a little bit better.  (We never said our help was free, we 
just said it didn't cost _money_.  Ok, the FSF did but they don't speak for 
all of us...)

> Very often, product developers are several versions behind, with
> no easy way to use the current kernel version.

I'm aware of that.  But if you can't set up a test system to reproduce the bug 
on a current system, the rest of us haven't got a _chance_.

> For example, a 
> common scenario is starting with a kernel that comes with a board
> (with source mind you), where the kernel came from the semi-conductor
> vendor, who paid a Linux vendor to do a port, and it was
> released in a time-frame relative to the Linux vendor's
> product schedule.

Then poke your vendor to fix the problem.

If you've decided to use a divergent fork from a vendor rather than the 
mainstream version, then the vendor has to support that fork for you because 
we're not going to be familiar with it.  (You can _hire_ one of us to support 
it for you, but we're not going to do so on a volunteer basis.)

We're happy to debug _our_ code.  But "our code" is the current vanilla 
release tarball.  If you can't reproduce the problem in the current vanilla 
tarball, then it's not our bug.  If you can only reproduce it in an older 
version: congratulations, we must have fixed it since.  If you can only 
reproduce it in some other fork, obviously their changes introduced the bug.  
If it's "your code plus this patch", we need to see the patch.

If _you_ can't reproduce it in our code, how do you expect _us_ to?

> This is how you end up having people STARTING projects today
> using a 2.6.11 kernel.  (I know of many).

Oldest I've seen a new project launch with this year is 2.6.15, but I agree 
with your point.

Whoever decided backporting bug fixes to a 2.6.16 kernel forever was a good 
idea seems to have muddied the waters a bit.  Ironically I don't know anybody 
actually _using_ that version, but I've seen several people point to it to 
show that "the community" supports arbitrarily older versions forever, and 
thus they don't have to upgrade to get support, and 2.6.18 is actually 
_newer_ than that...

> The real difficulty, when a developer finds themselves in
> this position, is how to forward-port the BSP code necessary to
> reproduce the bug in the current kernel.  Often, the code
> is not isolated well enough (this is a vendor problem that
> really needs attention.  If you have the BSP in patches, it
> is usually not too bad to forward port even across several
> kernel versions.  But many vendors don't ship stuff this way.)

Yup.  Sucks, doesn't it?  This is not a problem that improves with the passage 
of time.

Might be a good idea to make it clear up front that even if your changes never 
get mainlined, failure to break up and break out your patches is still likely 
to cause maintenance problems down the road.

> The fact is, that by a series of small steps and delays by
> the linux vendor, chip vendor, board vendor,
> and product developer the code is out-of step.

Hence the importance of breaking out and breaking up the changes.

> It's easy to say "don't get in this position", but
> this even happens when everyone is playing nice and actively
> trying to mainline stuff.  BSP support in arch trees often
> lag mainline by a version or two.

Getting out of sync is inevitable.  Happens to full-time kernel developers, 
that's why they have their own trees.  That's a separate issue from asking 
for patches and getting a source tarball that compiles instead.  "Here's a 
haystack, find the needle."

Mainlining changes and breaking them up into clean patches on top of some 
vanilla version (_any_ vanilla version) are two separate things.  You have to 
win one battle before you can even start the other.

> The number of parties involved here is why, IMHO, it has
> taken so long to make improvements in this area.

The lack of a clear consistent message from us to the vendors hasn't helped.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Kernel boot problem on IXP422 Rev. A

[Marcus Tangermann]

Hello Greg,

> Kernel dying, not just no console?
To be honest I don't know ;-) Maybe it's only an issue with the console. 
It should be ttyS0 as this works with 2.6.19-uc1. The snapgear kernel 
works when I use the settings for the MonteJade board.
Today I took a look at the sources of the snapgear kernel where the 
CONFIG_MARCH_MONTEJADE is used, and there are some modifications within 
the 8250 serial driver for this board.  I will take a closer look at it 
tomorrow.

Best regards
Marcus

Re: Kernel boot problem on IXP422 Rev. A

[Greg Ungerer]

Hi Marcus,

Marcus Tangermann wrote:
>> Kernel dying, not just no console?
> To be honest I don't know ;-) Maybe it's only an issue with the console. 
> It should be ttyS0 as this works with 2.6.19-uc1. The snapgear kernel 
> works when I use the settings for the MonteJade board.
> Today I took a look at the sources of the snapgear kernel where the 
> CONFIG_MARCH_MONTEJADE is used, and there are some modifications within 
> the 8250 serial driver for this board.  I will take a closer look at it 
> tomorrow.

Over the years there have been some changes to the serial
setup on those boards. Some setups leave out the first
serial port definition, and so what is otherwise considered
ttyS1 will be ttyS0. Something to look out for anyway.

Regards
Greg



------------------------------------------------------------------------
Greg Ungerer  --  Chief Software Dude       EMAIL:     gerg@snapgear.com
Secure Computing Corporation                PHONE:       +61 7 3435 2888
825 Stanley St,                             FAX:         +61 7 3891 3630
Woolloongabba, QLD, 4102, Australia         WEB: http://www.SnapGear.com