[U-Boot] Help command broken on AT91SAM9G10-EK

[U-Boot] Help command broken on AT91SAM9G10-EK

[Justin Waters]

It looks like the "help" command is broken on the AT91SAM9G10-EK on
2009.11-rc1.  I get the following message any time I try to run it:

Unknown command 'help' - try 'help'

I've bisected the issue, and it looks like it started with the addition
of the "editenv" command (commit
246c69225c7b962d5c93e92282b78ca9fc5fefee).  

After some further digging, I found some weirdness with the help command
beginning with the lead-up work to the editenv commit. Strange
characters where being printed to the console whenever the help command
was run.  This issue started with commit
ecc5500ee487170d8af6ff893fd1e0082380a01a, readline(): Add ability to
modify a string buffer.

I've also tested the AT91SAM9263-EK, and everything looks great.  I
believe the difference is related to the fact that the 9G10 does not use
hush, while the 9263 does. Indeed, if you look at the main loop,
non-hush systems use "readline" (which is indirectly modified by the
aforementioned commits), while hush systems do not.

I'm not really sure what would be the best way to fix this.  Any ideas?

-Justin Waters

[U-Boot] Help command broken on AT91SAM9G10-EK

[Tom]

Justin Waters wrote:
> It looks like the "help" command is broken on the AT91SAM9G10-EK on
> 2009.11-rc1.  I get the following message any time I try to run it:
> 
> Unknown command 'help' - try 'help'
> 
> I've bisected the issue, and it looks like it started with the addition
> of the "editenv" command (commit
> 246c69225c7b962d5c93e92282b78ca9fc5fefee).  
> 
> After some further digging, I found some weirdness with the help command
> beginning with the lead-up work to the editenv commit. Strange
> characters where being printed to the console whenever the help command
> was run.  This issue started with commit
> ecc5500ee487170d8af6ff893fd1e0082380a01a, readline(): Add ability to
> modify a string buffer.
> 
> I've also tested the AT91SAM9263-EK, and everything looks great.  I
> believe the difference is related to the fact that the 9G10 does not use
> hush, while the 9263 does. Indeed, if you look at the main loop,
> non-hush systems use "readline" (which is indirectly modified by the
> aforementioned commits), while hush systems do not.
> 
> I'm not really sure what would be the best way to fix this.  Any ideas?
> 

What is the impact of just reverting the commits ?
Tom

> -Justin Waters
> 
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] Help command broken on AT91SAM9G10-EK

[Peter Tyser]

On Fri, 2009-11-13 at 11:10 -0500, Justin Waters wrote:
> It looks like the "help" command is broken on the AT91SAM9G10-EK on
> 2009.11-rc1.  I get the following message any time I try to run it:
> 
> Unknown command 'help' - try 'help'
> 
> I've bisected the issue, and it looks like it started with the addition
> of the "editenv" command (commit
> 246c69225c7b962d5c93e92282b78ca9fc5fefee).  
> 
> After some further digging, I found some weirdness with the help command
> beginning with the lead-up work to the editenv commit. Strange
> characters where being printed to the console whenever the help command
> was run.  This issue started with commit
> ecc5500ee487170d8af6ff893fd1e0082380a01a, readline(): Add ability to
> modify a string buffer.
> 
> I've also tested the AT91SAM9263-EK, and everything looks great.  I
> believe the difference is related to the fact that the 9G10 does not use
> hush, while the 9263 does. Indeed, if you look at the main loop,
> non-hush systems use "readline" (which is indirectly modified by the
> aforementioned commits), while hush systems do not.
> 
> I'm not really sure what would be the best way to fix this.  Any ideas?

Hi Justin,
Thanks for reporting the issue.  I tested the changes with/without Hush
when I submitted the patches and the current top-of-tree U-Boot code
seems to work on my XPedite5370 (MPC8572-based) board here without Hush
enabled.

Do no commands work for you, or just "help" in particular?

And are you seeing 2 separate issues: 1 where the help output is
garbled, and 1 where the "help" command can't be ran?

Would it be possible for you to add a bit of debug to diagnose the
problem further as I can't seem to reproduce it here?

Thanks,
Peter

[U-Boot] Help command broken on AT91SAM9G10-EK

[Justin Waters]

Hi Peter,

On Fri, 2009-11-13 at 11:50 -0500, Peter Tyser wrote:
> Do no commands work for you, or just "help" in particular?

From what I can tell, just "help".

> And are you seeing 2 separate issues: 1 where the help output is
> garbled, and 1 where the "help" command can't be ran?

Yes, I am seeing two separate issues.  It depends on which commit I use.
If I go back to the "readline" commit, then I get garbled output.  Once
the editenv commit is added back, I get the 'help not found' error.

If I disable editenv and revert the changes to 'cread_line', then the
error disappears.  It will also work if I disable LONG_HELP and revert
cread_line.

> Would it be possible for you to add a bit of debug to diagnose the
> problem further as I can't seem to reproduce it here?

Absolutely.  What would you like me to check?

> Thanks,
> Peter

-Justin

[U-Boot] Help command broken on AT91SAM9G10-EK

[Peter Tyser]

On Fri, 2009-11-13 at 12:01 -0500, Justin Waters wrote:
> Hi Peter,
> 
> On Fri, 2009-11-13 at 11:50 -0500, Peter Tyser wrote:
> > Do no commands work for you, or just "help" in particular?
> 
> >From what I can tell, just "help".

Interesting...

> > And are you seeing 2 separate issues: 1 where the help output is
> > garbled, and 1 where the "help" command can't be ran?
> 
> Yes, I am seeing two separate issues.  It depends on which commit I use.
> If I go back to the "readline" commit, then I get garbled output.  Once
> the editenv commit is added back, I get the 'help not found' error.

I'm not sure what to make of this.  I can't figure out how
"246c69225c7b962d5c93e92282b78ca9fc5fefee - Add 'editenv' command" would
fix the garbled output or cause the help command not to be found.  Its
just adding a new command and shouldn't really have any impact on common
code.

> If I disable editenv and revert the changes to 'cread_line', then the
> error disappears.  It will also work if I disable LONG_HELP and revert
> cread_line.
> 
> > Would it be possible for you to add a bit of debug to diagnose the
> > problem further as I can't seem to reproduce it here?
> 
> Absolutely.  What would you like me to check?

Thanks.  If I understand correctly, chronologically you first start
having issues when "ecc5500ee487170d8af6ff893fd1e0082380a01a -
readline(): Add ability to modify a string buffer" is applied.

If this is the case, I'd try debugging this issue first before
addressing the later 'help not found' issue.

Is there any pattern in the garbled output? Could you send an example of
it?  Is only the output of 'help' is garbled, everything else is fine?

Could you add some debug to print out the value of init_len in
common/main.c:cread_line()?  In theory as long as you don't run
"editenv" it should always be 0.  Assuming it is always 0, could you try
commenting out the following snippet in the same file:

      if (init_len)
               cread_add_str(buf, init_len, 1, &num, &eol_num, buf, *len);

Then also try commenting out the modification to
common/main.c:readline():

      console_buffer[0] = '\0';

Those 2 modifications are all that commit
ecc5500ee487170d8af6ff893fd1e0082380a01a added - I'd be curious to know
which one caused the problem.


Another interesting data point would be to remove CONFIG_SYS_HUSH_PARSER
from the AT91SAM9263-EK u-boot image and see if it breaks in the same
way.

Thanks again,
Peter

[U-Boot] Help command broken on AT91SAM9G10-EK

[Justin Waters]

On Fri, 2009-11-13 at 12:50 -0500, Peter Tyser wrote:
> On Fri, 2009-11-13 at 12:01 -0500, Justin Waters wrote:
> > Hi Peter,
> > 
> > On Fri, 2009-11-13 at 11:50 -0500, Peter Tyser wrote:
> > > Do no commands work for you, or just "help" in particular?
> > 
> > >From what I can tell, just "help".
> 
> Interesting...

I haven't noticed any other issues, but that doesn't mean there aren't
any ;)

> > > And are you seeing 2 separate issues: 1 where the help output is
> > > garbled, and 1 where the "help" command can't be ran?
> > 
> > Yes, I am seeing two separate issues.  It depends on which commit I use.
> > If I go back to the "readline" commit, then I get garbled output.  Once
> > the editenv commit is added back, I get the 'help not found' error.
> 
> I'm not sure what to make of this.  I can't figure out how
> "246c69225c7b962d5c93e92282b78ca9fc5fefee - Add 'editenv' command" would
> fix the garbled output or cause the help command not to be found.  Its
> just adding a new command and shouldn't really have any impact on common
> code.

Well, editenv doesn't fix the garbled output, it just makes it so you
can't see it.  The help command issue is what really confuses me.  If I
just disable the editenv command, running "help" seems to lock up the
board.  That seems really weird to me, since it seems to disable all of
the code that was added by the previous patch.  I'm going to dig into
this a little more.  If I also revert the changes in cread_line,
everything works fine.

> Thanks.  If I understand correctly, chronologically you first start
> having issues when "ecc5500ee487170d8af6ff893fd1e0082380a01a -
> readline(): Add ability to modify a string buffer" is applied.

Yes, that's correct.

> If this is the case, I'd try debugging this issue first before
> addressing the later 'help not found' issue.

I agree.  That's the tact that I was trying to take before.

> Is there any pattern in the garbled output? Could you send an example of
> it?  Is only the output of 'help' is garbled, everything else is fine?

So there are three commits in a row:

ecc5500 - readline(): Add ability to modify a string buffer
b0fa8e5 - setenv(): Delete 0-length environment variables
246c692 - Add 'editenv' command

After the first commit, the garbled text appears at the end of help.
After the second, running *help* hangs the board.
After the third, I get the 'help not found' message.  Although, now that
I try to reproduce the error, it seems to hang as well.  Possibly
something still sitting around in memory that was causing the problem?

> Could you add some debug to print out the value of init_len in
> common/main.c:cread_line()?  In theory as long as you don't run
> "editenv" it should always be 0.  Assuming it is always 0, could you try
> commenting out the following snippet in the same file:
> 
>       if (init_len)
>                cread_add_str(buf, init_len, 1, &num, &eol_num, buf, *len);
> 
> Then also try commenting out the modification to
> common/main.c:readline():
> 
>       console_buffer[0] = '\0';
> 
> Those 2 modifications are all that commit
> ecc5500ee487170d8af6ff893fd1e0082380a01a added - I'd be curious to know
> which one caused the problem.

Definitely the first one.  I can leave the second one in without a
problem.

> 
> Another interesting data point would be to remove CONFIG_SYS_HUSH_PARSER
> from the AT91SAM9263-EK u-boot image and see if it breaks in the same
> way.

Now things have gotten very interesting... SAM9263 works great without
hush, but the 9G10 and 9261 (which use the same base code, just slightly
different configurations) are completely broken.  Also, I can't add hush
into these boards, or they won't boot.  This error that I see after
applying your patches seems likely to be a symptom of a larger issue
with these two boards.  I'm going to try some of the other Atmel boards
that I have lying around (9260, 9G20).  I'll let you know what I find
out.

> Thanks again,
> Peter

Thanks,

Justin

[U-Boot] Help command broken on AT91SAM9G10-EK

[Peter Tyser]

Hi Justin,

> > > > And are you seeing 2 separate issues: 1 where the help output is
> > > > garbled, and 1 where the "help" command can't be ran?
> > > 
> > > Yes, I am seeing two separate issues.  It depends on which commit I use.
> > > If I go back to the "readline" commit, then I get garbled output.  Once
> > > the editenv commit is added back, I get the 'help not found' error.
> > 
> > I'm not sure what to make of this.  I can't figure out how
> > "246c69225c7b962d5c93e92282b78ca9fc5fefee - Add 'editenv' command" would
> > fix the garbled output or cause the help command not to be found.  Its
> > just adding a new command and shouldn't really have any impact on common
> > code.
> 
> Well, editenv doesn't fix the garbled output, it just makes it so you
> can't see it.  The help command issue is what really confuses me.  If I
> just disable the editenv command, running "help" seems to lock up the
> board.  That seems really weird to me, since it seems to disable all of
> the code that was added by the previous patch.  I'm going to dig into
> this a little more.  If I also revert the changes in cread_line,
> everything works fine.
> 
> > Thanks.  If I understand correctly, chronologically you first start
> > having issues when "ecc5500ee487170d8af6ff893fd1e0082380a01a -
> > readline(): Add ability to modify a string buffer" is applied.
> 
> Yes, that's correct.
> 
> > If this is the case, I'd try debugging this issue first before
> > addressing the later 'help not found' issue.
> 
> I agree.  That's the tact that I was trying to take before.
> 
> > Is there any pattern in the garbled output? Could you send an example of
> > it?  Is only the output of 'help' is garbled, everything else is fine?
> 
> So there are three commits in a row:
> 
> ecc5500 - readline(): Add ability to modify a string buffer
> b0fa8e5 - setenv(): Delete 0-length environment variables
> 246c692 - Add 'editenv' command
> 
> After the first commit, the garbled text appears at the end of help.
> After the second, running *help* hangs the board.
> After the third, I get the 'help not found' message.  Although, now that
> I try to reproduce the error, it seems to hang as well.  Possibly
> something still sitting around in memory that was causing the problem?
> 
> > Could you add some debug to print out the value of init_len in
> > common/main.c:cread_line()?  In theory as long as you don't run
> > "editenv" it should always be 0.  Assuming it is always 0, could you try
> > commenting out the following snippet in the same file:
> > 
> >       if (init_len)
> >                cread_add_str(buf, init_len, 1, &num, &eol_num, buf, *len);
> > 
> > Then also try commenting out the modification to
> > common/main.c:readline():
> > 
> >       console_buffer[0] = '\0';
> > 
> > Those 2 modifications are all that commit
> > ecc5500ee487170d8af6ff893fd1e0082380a01a added - I'd be curious to know
> > which one caused the problem.
> 
> Definitely the first one.  I can leave the second one in without a
> problem.

What is the value of "init_len"?  It should always be 0 other than when
'editenv' is called.  So I doubt the added code is ever called.

> > 
> > Another interesting data point would be to remove CONFIG_SYS_HUSH_PARSER
> > from the AT91SAM9263-EK u-boot image and see if it breaks in the same
> > way.
> 
> Now things have gotten very interesting... SAM9263 works great without
> hush, but the 9G10 and 9261 (which use the same base code, just slightly
> different configurations) are completely broken.  Also, I can't add hush
> into these boards, or they won't boot.  This error that I see after
> applying your patches seems likely to be a symptom of a larger issue
> with these two boards.  I'm going to try some of the other Atmel boards
> that I have lying around (9260, 9G20).  I'll let you know what I find
> out.

Thanks for the info.

My guess is that somehow the end of u-boot command section is getting
corrupted.  Perhaps the editenv patches, etc increase the size of your
image such there's an overlap in sections that isn't reported as an
error?  Or is the stack above the U-boot image in RAM?  Or malloc pool,
etc?

If I look at my System.map file I see this at the end:
<lots more __u_boot_cmds>
fffd2bbc D __u_boot_cmd_test
fffd2bd4 D __u_boot_cmd_exit
fffd2bec D __u_boot_cmd_help
fffd2c04 D __u_boot_cmd_question_mark
fffd2c1c D __u_boot_cmd_showvar
fffd2c34 A __start___ex_table
fffd2c34 A __stop___ex_table
fffd2c34 A __u_boot_cmd_end
fffd2d00 A __init_begin
fffd2d00 A __init_end
fffff000 T _start_e500
fffff18c t switch_as

If you look at other commands near help (eg exit or test) in your
System.map file, do they have the same issues as as 'help'?  Seems fishy
that the u-boot command table is near the end of the u-boot image, and
you're seeing issues specifically with the end of the u-boot command
table.

Best,
Peter

[U-Boot] Help command broken on AT91SAM9G10-EK

[Justin Waters]

Peter,

On Fri, 2009-11-13 at 16:39 -0500, Peter Tyser wrote:
> My guess is that somehow the end of u-boot command section is getting
> corrupted.  Perhaps the editenv patches, etc increase the size of your
> image such there's an overlap in sections that isn't reported as an
> error?  Or is the stack above the U-boot image in RAM?  Or malloc pool,
> etc?
> If you look at other commands near help (eg exit or test) in your
> System.map file, do they have the same issues as as 'help'?  Seems fishy
> that the u-boot command table is near the end of the u-boot image, and
> you're seeing issues specifically with the end of the u-boot command
> table.
> 

Brilliant.  I removed a couple of commands (USB, in particular), and it
seems to be working fine.  No garbled messages, no system halt.

I'll look into this some more on Monday.  It seems to only affect the
9261/9G10 family.  Everything works fine on the 9G20 and 9263.

Thank you for your help!  editenv is a really nice addition, by the way.

> Best,
> Peter

Thanks,
Justin

[U-Boot] Help command broken on AT91SAM9G10-EK

[Justin Waters]

Peter,

On Fri, 2009-11-13 at 17:10 -0500, Justin Waters wrote:
> Peter,
> 
> On Fri, 2009-11-13 at 16:39 -0500, Peter Tyser wrote:
> > My guess is that somehow the end of u-boot command section is getting
> > corrupted.  Perhaps the editenv patches, etc increase the size of your
> > image such there's an overlap in sections that isn't reported as an
> > error?  Or is the stack above the U-boot image in RAM?  Or malloc pool,
> > etc?
> > If you look at other commands near help (eg exit or test) in your
> > System.map file, do they have the same issues as as 'help'?  Seems fishy
> > that the u-boot command table is near the end of the u-boot image, and
> > you're seeing issues specifically with the end of the u-boot command
> > table.
> > 
> 
> Brilliant.  I removed a couple of commands (USB, in particular), and it
> seems to be working fine.  No garbled messages, no system halt.
> 
> I'll look into this some more on Monday.  It seems to only affect the
> 9261/9G10 family.  Everything works fine on the 9G20 and 9263.
> 
> Thank you for your help!  editenv is a really nice addition, by the way.
> 
> > Best,
> > Peter
> 
> Thanks,
> Justin

Got it.  The problem was with the first stage bootloader, which was
apparently only copying the first 0x33900 bytes of U-Boot into RAM.  The
image containing the editenv changes apparently broke this barrier,
causing the image size to jump up to 0x33A00 (thanks to page alignment).
After changing the value to something larger in AT91 Bootstrap (I picked
0x33C00 since it's a multiple of the page size of the flash), everything
works fine.

I didn't encounter the problem on the 9G20 because it's a smaller image
(no LCD driver), and the 9263 was loading from NAND flash instead of
dataflash.

Thanks again!

-Justin