Ethernet in a cold climate / SMDK6410

Ethernet in a cold climate / SMDK6410

[Andy Green]

Hi -

I broke out my old SMDK6410 last night and tried linux-next then Ben's 
git on it (currently next-s3c64xx-updates) started from s3c6410 Qi / SD 
Card boot.

It's basically happy but OTG doesn't seem workable for Ethernet gadget. 
  Well it's dumping a lot of debug messages so I guess maybe that's 
where it got up to, fair enough.

I eyeballed the back of the SMDK and I see a CS8900A and a SMSC LAN9115 
both sitting there, coming out on separate RJ45s.  When I look at the 
machine init file, I see Mark added

static struct resource smdk6410_smsc911x_resources[] = {
	[0] = {
		.start = 0x18000000,
		.end   = 0x18000000 + SZ_64K - 1,
		.flags = IORESOURCE_MEM,
	},
...

What is the deal with that in terms of SDMK settings and assumptions 
about bootloader prepping nCS signals?  SMDK docs I have let me twiddle 
switches and get "ethernet" connected to various selects.  What does 
0x18000000 represent and how should the board be set so CS8900A doesn't 
crash the party?

After some debug-by-printk, I see smsc911x probe is failing with:

smsc911x: BYTE_TEST: 0xFFFFFFFF
smsc911x: top 16 bits equal to bottom 16 bits
smsc911x: This may mean the chip is set for 32 bit while the bus is 
reading 16 bit
smsc911x: init failed

So I am guessing the nCS pin is not configured, something is missing a 
clock... one of the dozens of mini DIP switches is "wrong"?

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Mon, Dec 28, 2009 at 06:40:00PM +0000, Andy Green wrote:

> What is the deal with that in terms of SDMK settings and assumptions
> about bootloader prepping nCS signals?  SMDK docs I have let me
> twiddle switches and get "ethernet" connected to various selects.
> What does 0x18000000 represent and how should the board be set so
> CS8900A doesn't crash the party?

I've never used the board with the CS8900A enabled, the bootloaders I've
seen appear to support the SMSC chip only.

The memory address was lifted from the Samsung BSP code IIRC.

Ethernet in a cold climate / SMDK6410

[Andy Green]

On 12/28/09 19:00, Somebody in the thread at some point said:

Hi Mark -

>> What is the deal with that in terms of SDMK settings and assumptions
>> about bootloader prepping nCS signals?  SMDK docs I have let me
>> twiddle switches and get "ethernet" connected to various selects.
>> What does 0x18000000 represent and how should the board be set so
>> CS8900A doesn't crash the party?
>
> I've never used the board with the CS8900A enabled, the bootloaders I've
> seen appear to support the SMSC chip only.
>
> The memory address was lifted from the Samsung BSP code IIRC.

Thanks for the quick reply.

It's fine if I get the SMSC chip happy, I just mention the CS8900A 
because it is there are the docs only talk about "ethernet" so I expect 
conflict.

So what was your setup when Ethernet worked OK...

  - the Samsung U-Boot fork that came with the SMDK?

  - magic address 0x18000000 OK

  - 60+ minidip config was what?

I'm guessing 0x18000000 is on a particular nCS signal that needs 
configuring (should that really not be the business of the machine 
definition file?) and the mini DIPs setting to match.

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Mon, Dec 28, 2009 at 07:07:15PM +0000, Andy Green wrote:

> So what was your setup when Ethernet worked OK...

>  - the Samsung U-Boot fork that came with the SMDK?

Yes, whatever was flashed on it when Samsung last had the board.  The
bootloader does explicitly say it's looking for the SMSC chip when
setting it up.

>  - magic address 0x18000000 OK

Yes.

>  - 60+ minidip config was what?

Factory default - I can't immediately visualise which switches you're
talking about and I'm out of the office until next Monday so I can't
check with the boards how they're set up.  I don't recall any relevant
switches.

> I'm guessing 0x18000000 is on a particular nCS signal that needs
> configuring (should that really not be the business of the machine
> definition file?) and the mini DIPs setting to match.

I'd expect so.  

I'm not sure what you mean by the "machine definition file" here?  The
large number of non-probable and incompatible changes on the Samsung
boards mean that it's generally best to leave them at factory default
settings unless you have a particular need to change things since the
kernel tends to be configured for the default setup.

Ethernet in a cold climate / SMDK6410

[Andy Green]

On 12/28/09 19:22, Somebody in the thread at some point said:

Hi -

>>   - the Samsung U-Boot fork that came with the SMDK?
>
> Yes, whatever was flashed on it when Samsung last had the board.  The
> bootloader does explicitly say it's looking for the SMSC chip when
> setting it up.

I went and looked at the sources for that I got from Samsung, it talks 
only about CS8900.

#define CS8900_Tacs	(0x0)	// 0clk		address set-up
#define CS8900_Tcos	(0x4)	// 4clk		chip selection set-up
#define CS8900_Tacc	(0xE)	// 14clk	access cycle
#define CS8900_Tcoh	(0x1)	// 1clk		chip selection hold
#define CS8900_Tah	(0x4)	// 4clk		address holding time
#define CS8900_Tacp	(0x6)	// 6clk		page mode access cycle
#define CS8900_PMC	(0x0)	// normal(1data)page mode configuration

...

	SROM_BW_REG &= ~(0xf << 4);
	SROM_BW_REG |= (1<<7) | (1<<6) | (1<<4);
	SROM_BC1_REG = 
((CS8900_Tacs<<28)+(CS8900_Tcos<<24)+(CS8900_Tacc<<16)+(CS8900_Tcoh<<12)+(CS8900_Tah<<8)+(CS8900_Tacp<<4)+(CS8900_PMC));

I had a quick grep for corresponding register defines in S3C64xx Linux 
bits but didn't see anything.

>>   - 60+ minidip config was what?
>
> Factory default - I can't immediately visualise which switches you're
> talking about and I'm out of the office until next Monday so I can't
> check with the boards how they're set up.  I don't recall any relevant
> switches.

No worries.  There are a row of mainly 4-switch mini-dips under the LCD 
that control who gets to see which chip select, I'm pretty sure setting 
them wrong will break access to "ethernet" whatever that means.

If we had like S3C6410_PHYS_NCS2 instead of 0x18000000 I could get a 
clue, but I realize you didn't get that from the Samsung tree either.

>> I'm guessing 0x18000000 is on a particular nCS signal that needs
>> configuring (should that really not be the business of the machine
>> definition file?) and the mini DIPs setting to match.
>
> I'd expect so.
>
> I'm not sure what you mean by the "machine definition file" here?  The

I meant mach-smdk6410.c, since it wants to configure Linux to use stuff 
that has dependencies it doesn't control.  If it's going to claim 
something is there at 0x18000000 for a particular machine then it's at 
least arguable it should configure any soft prerequisites (nCS) 
accordingly and document the ones out of its control.

> large number of non-probable and incompatible changes on the Samsung
> boards mean that it's generally best to leave them at factory default
> settings unless you have a particular need to change things since the
> kernel tends to be configured for the default setup.

Yeah.  But again if the successful operation of the file that defines 
how Linux works with the SMDK is dependent on stuff like magic switch 
configuration ideally it would tell what the dependencies are in a 
comment, accepting random changes to boards you don't have are out of 
scope.  That one file anyway mach-smdk6410.c is super-specific to the 
board and is the only place to talk about the dependencies.

I understand that you might be able to get that working and yet not 
understand the intricacies of the dev board, and we're still ahead that 
you managed to get anything working.

My board is as shipped with the switches AFAIK except configured for SD 
Card boot according to Samsung docs.

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Mon, Dec 28, 2009 at 07:45:07PM +0000, Andy Green wrote:
> On 12/28/09 19:22, Somebody in the thread at some point said:

>> Yes, whatever was flashed on it when Samsung last had the board.  The
>> bootloader does explicitly say it's looking for the SMSC chip when
>> setting it up.

> I went and looked at the sources for that I got from Samsung, it talks  
> only about CS8900.

That's...  interesting.  To be honest I don't think I've ever seen the
source code for the bootloaders, but none of the SMDK boards I've had
have defaulted to using the CS8900 - I've never actually seen software
that talked to it to my knowledge, the Samsung bootloaders and kernels
were all using the SMSC chip and obviously that's what mainline has been
doing too.

>> I'm not sure what you mean by the "machine definition file" here?  The

> I meant mach-smdk6410.c, since it wants to configure Linux to use stuff  
> that has dependencies it doesn't control.  If it's going to claim  
> something is there at 0x18000000 for a particular machine then it's at  
> least arguable it should configure any soft prerequisites (nCS)  
> accordingly and document the ones out of its control.

Oh, sure - you were saying it should configure stuff and obviously most
of these things are completely out of scope for software which wasn't
very clear at all.  I quite frankly wasn't aware that there were any
relevant soft controls at all.

By the way, the other thing to watch out for with the Samsung boards is
that as well as the DIP switches there's a bunch of resistor fit options
that can be changed as well.

> I understand that you might be able to get that working and yet not  
> understand the intricacies of the dev board, and we're still ahead that  
> you managed to get anything working.

Right, and hopefully this is the sort of thing that Samsung are going to
be doing with their mainline work.

> My board is as shipped with the switches AFAIK except configured for SD  
> Card boot according to Samsung docs.

It sounds like you've at least got a completely different bootloader (or
the source is highly misleading).  The other thing to check is which rev
of the board you have, there have been several revisions though I don't
recall any changes in the networking.

Ethernet in a cold climate / SMDK6410

[Andy Green]

On 12/28/09 20:21, Somebody in the thread at some point said:

Hi -

>> I went and looked at the sources for that I got from Samsung, it talks
>> only about CS8900.
>
> That's...  interesting.  To be honest I don't think I've ever seen the
> source code for the bootloaders, but none of the SMDK boards I've had
> have defaulted to using the CS8900 - I've never actually seen software
> that talked to it to my knowledge, the Samsung bootloaders and kernels
> were all using the SMSC chip and obviously that's what mainline has been
> doing too.

Well it's a fact.  I sent you a link these GPL'd sources separately.

>> I meant mach-smdk6410.c, since it wants to configure Linux to use stuff
>> that has dependencies it doesn't control.  If it's going to claim
>> something is there at 0x18000000 for a particular machine then it's at
>> least arguable it should configure any soft prerequisites (nCS)
>> accordingly and document the ones out of its control.
>
> Oh, sure - you were saying it should configure stuff and obviously most
> of these things are completely out of scope for software which wasn't
> very clear at all.  I quite frankly wasn't aware that there were any
> relevant soft controls at all.

I would suggest that relying on (mysterious, changeable) bootloader to 
set up stuff like nCS timing and which nCS would be a dependency that 
would be good to avoid.  People are trying to use the dev board as a 
basis for their own designs and they have to ship an actual 
non-quantum-mist bootloader with that.

> By the way, the other thing to watch out for with the Samsung boards is
> that as well as the DIP switches there's a bunch of resistor fit options
> that can be changed as well.

Yeah, I saw it on SPI0 before (I want to use this as an SPI peripheral 
meddling platform at the moment).  The socket is sitting there with NC 
resistor pack so no signals.

> It sounds like you've at least got a completely different bootloader (or
> the source is highly misleading).  The other thing to check is which rev
> of the board you have, there have been several revisions though I don't
> recall any changes in the networking.

Right I wrote Qi bootloader support for s3c6410.  This allows true SD 
boot from SMDK.  "True SD boot" means that the bootloader itself is on 
SD Card and is brought over into RAM by the CPU ROM, so there is nothing 
to brick on the device itself.

If anyone is interested see the txtr branch of

  http://git.warmcat.com/cgi-bin/cgit/qi/

SMDK6410 docs tell the following to enable SD Card boot:

CFGB1 - CFG4 (under the LCD)

~____ __~_ ~___ ____ ____ ____ ~_ ~~~_ __~_ ____ ____

CFG3

[
[
[
[
[
  ]

CFG4

  ]
  ]
[
[

CFG1

  ]
[
[
[

J8: 1-2
J7: 1-2
J6: 1-2

You also need a particular ROM revision in your SMDK but this one is 
over a year old so I guess it's common now.

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Mon, Dec 28, 2009 at 08:46:57PM +0000, Andy Green wrote:

> I would suggest that relying on (mysterious, changeable) bootloader to  
> set up stuff like nCS timing and which nCS would be a dependency that  
> would be good to avoid.  People are trying to use the dev board as a  
> basis for their own designs and they have to ship an actual  
> non-quantum-mist bootloader with that.

Like I say, I wasn't aware that these were soft configurable in the
first place - I'd never needed to look beyond the hookup of the device
in the Samsung kernel plus some detective work with the IRQ polarity
configuration in the driver since the board miswires it.  I do agree
that if these things are software configurable then we ought to be doing
it in the kernel, I strongly suspect that any setup done by the
bootloader is only being done when booting over the network.

>> It sounds like you've at least got a completely different bootloader (or
>> the source is highly misleading).  The other thing to check is which rev
>> of the board you have, there have been several revisions though I don't
>> recall any changes in the networking.

> Right I wrote Qi bootloader support for s3c6410.  This allows true SD  
> boot from SMDK.  "True SD boot" means that the bootloader itself is on  
> SD Card and is brought over into RAM by the CPU ROM, so there is nothing  
> to brick on the device itself.

Qi would be a massive step back for me in terms of usability
unfortunately - having to transfer things to an SD card to boot them
would add a lot to my edit, compile run cycle compared to netboot.

Ethernet in a cold climate / SMDK6410

[Andy Green]

On 12/28/09 22:44, Somebody in the thread at some point said:

Hi Mark -

>> I would suggest that relying on (mysterious, changeable) bootloader to
>> set up stuff like nCS timing and which nCS would be a dependency that
>> would be good to avoid.  People are trying to use the dev board as a
>> basis for their own designs and they have to ship an actual
>> non-quantum-mist bootloader with that.
>
> Like I say, I wasn't aware that these were soft configurable in the
> first place - I'd never needed to look beyond the hookup of the device
> in the Samsung kernel plus some detective work with the IRQ polarity
> configuration in the driver since the board miswires it.  I do agree
> that if these things are software configurable then we ought to be doing
> it in the kernel, I strongly suspect that any setup done by the
> bootloader is only being done when booting over the network.

Glad we agree on that.

Bootloaders are kind of out of control of these dev boards, iMX31 
LiteKIT even came with a closed source bootloader.  That destabilizes 
any certainty you can have about basing a design on the dev board, 
because you don't know how dependent successful behaviour will be on 
magic hidden in the bootloader, even if you got a fully working software 
stack on the dev board.

So I think the mach-blah.c files should try to assert the state of 
dependencies themselves for stuff they are saying they support, and make 
no assumption that the bootloader has done anything.  (And indeed I 
don't think the bootloaders should do anything except enough to get 
Linux started, but that's another story).

Like I say it sets up a magic nCS signal in the bootloader init, it 
talks about it as CS8900A but elsewhere they talk about "ethernet", so 
it may simply use this nCS to get to both ethernet chips, I didn't look 
at the schematic yet.

Just a sanity check, it was the case for you too that g_ether is broken 
and although you get a logical usb0 at both sides, they cannot communicate?

>> Right I wrote Qi bootloader support for s3c6410.  This allows true SD
>> boot from SMDK.  "True SD boot" means that the bootloader itself is on
>> SD Card and is brought over into RAM by the CPU ROM, so there is nothing
>> to brick on the device itself.
>
> Qi would be a massive step back for me in terms of usability
> unfortunately - having to transfer things to an SD card to boot them
> would add a lot to my edit, compile run cycle compared to netboot.

You seem pretty sure about that, so much so I guess you never tried this 
workflow.

There are two cases with SD Card boot and kernel work:

1) you screwed your kernel and it blows chunks on boot.  In that case, 
you either have to make arrangements in the bootloader to check a button 
and use a backup kernel if pressed (Qi supports this generically), or 
pop the SD card and write the kernel on the host.  In the button case, 
you just reset with that button down and use a host build script to ssh 
/ rsync over your next kernel try without touching the SD Card.

I guess you see the above case more often than most because of what 
you're doing, but actually I find it's only when I meddle in particular 
areas I run that risk; most things fail to work nicely until fixed or 
OOPS in a way that you can still send over your fixes in that session.

2) you are working on a driver that can be done modular while there's 
risk of it blowing chunks.  In this case your build script can build the 
kernel on the host and update the module over ssh / rsync.  You then 
modprobe -r / modprobe it without touching the SD Card or even resetting.

I don't see those workflows as any "massive step back"; compared to raw 
NAND being able to pop and rewrite the SD card in an emergency is 
certainly a massive step forward.


There's another reason to not act special towards the device with stuff 
that does not represent what will ship, as the developer you're the 
first user of it and if you're not hammering on, experiencing and 
optimizing the boot flow that ships, nobody else will.

In addition if opposed to network boot case you are working on an SD 
image all the time that can be taken as the shipping image, not some 
special tree of files on a host whose speed of access and code paths to 
get to it are completely different.

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Tue, Dec 29, 2009 at 10:04:41AM +0000, Andy Green wrote:
> On 12/28/09 22:44, Somebody in the thread at some point said:

> Like I say it sets up a magic nCS signal in the bootloader init, it
> talks about it as CS8900A but elsewhere they talk about "ethernet",
> so it may simply use this nCS to get to both ethernet chips, I
> didn't look at the schematic yet.

I'd assume so given that the board has a physical switch selecting between
the two ethernet controllers.

> Just a sanity check, it was the case for you too that g_ether is
> broken and although you get a logical usb0 at both sides, they
> cannot communicate?

I've never used g_ether, and my use of USB has been limited to testing
power path management.

> >Qi would be a massive step back for me in terms of usability
> >unfortunately - having to transfer things to an SD card to boot them
> >would add a lot to my edit, compile run cycle compared to netboot.

> You seem pretty sure about that, so much so I guess you never tried
> this workflow.

Really, I've tried this and similar workflows.  Network boot beats
everything else I've tried.

> 1) you screwed your kernel and it blows chunks on boot.  In that
> case, you either have to make arrangements in the bootloader to
> check a button and use a backup kernel if pressed (Qi supports this
> generically), or pop the SD card and write the kernel on the host.
> In the button case, you just reset with that button down and use a
> host build script to ssh / rsync over your next kernel try without
> touching the SD Card.

Either physically swapping the SD card or having to dance with the
bootloader to revert to a known good configuration then refresh are very
involved relative to flipping the power switch and booting, partly
because they do actually take a relatively long time and partly because
they aren't the same routine normally used to boot and so require more
attention and thought (not much, but enough).

> 2) you are working on a driver that can be done modular while
> there's risk of it blowing chunks.  In this case your build script
> can build the kernel on the host and update the module over ssh /
> rsync.  You then modprobe -r / modprobe it without touching the SD
> Card or even resetting.

This is what I'm actually doing - things that can be built modular
generally are, but there's always things that won't allow that.

> I don't see those workflows as any "massive step back"; compared to
> raw NAND being able to pop and rewrite the SD card in an emergency
> is certainly a massive step forward.

Compared to NAND removable media can be a win, yes, though with fast
JTAG having to rewrite the flash needn't be any slower so it's not quite
that clear cut.

> There's another reason to not act special towards the device with
> stuff that does not represent what will ship, as the developer
> you're the first user of it and if you're not hammering on,
> experiencing and optimizing the boot flow that ships, nobody else
> will.

With what I do it is relatively rare for me to work directly on
production systems - the majority of the systems I'm using for driver
development involve lots of flying wires and never see the light of
anything except my desk, the end result being either core kernel changes
or chip-specific drivers.  Where I am working on production systems
generally either it's not possible to do anything except JTAG onto the
flash or the only thing changed by netboot is where the kernel image is
loaded from.

Ethernet in a cold climate / SMDK6410

[Andy Green]

On 12/29/09 12:33, Somebody in the thread at some point said:

Hi -

>> Like I say it sets up a magic nCS signal in the bootloader init, it
>> talks about it as CS8900A but elsewhere they talk about "ethernet",
>> so it may simply use this nCS to get to both ethernet chips, I
>> didn't look at the schematic yet.
>
> I'd assume so given that the board has a physical switch selecting between
> the two ethernet controllers.

Yeah I found that thismorning, CFG6.  I added a comment about that and 
the (discovered by trial and error) requirement that it's nCS1 
specifically needed to drive it to work with the 0x18000000 magic number.

I found that the nCS1 bus width defaults to 8-bit in the CPU, otherwise 
it's now addressable.  So I am adding some constants for s3c64xx SROM 
unit in the platform stuff and will set it up in mach-smdk6410.c.

>>> Qi would be a massive step back for me in terms of usability
>>> unfortunately - having to transfer things to an SD card to boot them
>>> would add a lot to my edit, compile run cycle compared to netboot.
>
>> You seem pretty sure about that, so much so I guess you never tried
>> this workflow.
>
> Really, I've tried this and similar workflows.  Network boot beats
> everything else I've tried.
>
>> 1) you screwed your kernel and it blows chunks on boot.  In that
>> case, you either have to make arrangements in the bootloader to
>> check a button and use a backup kernel if pressed (Qi supports this
>> generically), or pop the SD card and write the kernel on the host.
>> In the button case, you just reset with that button down and use a
>> host build script to ssh / rsync over your next kernel try without
>> touching the SD Card.
>
> Either physically swapping the SD card or having to dance with the
> bootloader to revert to a known good configuration then refresh are very
> involved relative to flipping the power switch and booting, partly
> because they do actually take a relatively long time and partly because
> they aren't the same routine normally used to boot and so require more
> attention and thought (not much, but enough).

Yeah.  All I can say is that constant testing of kernels that fail to 
boot is a somewhat specialized case.  You would be into pressing a 
button or popping the card.

>> 2) you are working on a driver that can be done modular while
>> there's risk of it blowing chunks.  In this case your build script
>> can build the kernel on the host and update the module over ssh /
>> rsync.  You then modprobe -r / modprobe it without touching the SD
>> Card or even resetting.
>
> This is what I'm actually doing - things that can be built modular
> generally are, but there's always things that won't allow that.

Sure.  In the ones you build modular, there's no cost to the SD flow.

>> I don't see those workflows as any "massive step back"; compared to
>> raw NAND being able to pop and rewrite the SD card in an emergency
>> is certainly a massive step forward.
>
> Compared to NAND removable media can be a win, yes, though with fast
> JTAG having to rewrite the flash needn't be any slower so it's not quite
> that clear cut.

Right, but if you can eliminate JTAG in the flow, especially at the 
factory, that is a major simplification.

SD + Qi has the advantage it eliminates completely "board bringup" at 
the factory.  Qi has no state stored on the board that changes boot 
flow.  So you can give a virgin board an SD and it brings itself up with 
no special gear or processing steps.

>> There's another reason to not act special towards the device with
>> stuff that does not represent what will ship, as the developer
>> you're the first user of it and if you're not hammering on,
>> experiencing and optimizing the boot flow that ships, nobody else
>> will.
>
> With what I do it is relatively rare for me to work directly on
> production systems - the majority of the systems I'm using for driver
> development involve lots of flying wires and never see the light of
> anything except my desk, the end result being either core kernel changes
> or chip-specific drivers.  Where I am working on production systems
> generally either it's not possible to do anything except JTAG onto the
> flash or the only thing changed by netboot is where the kernel image is
> loaded from.

Yeah understood.

Again all I can say is it's specialized case.  For normal dev work SD is 
either painless or hugely advantageous.

-Andy

Ethernet in a cold climate / SMDK6410

[Mark Brown]

On Tue, Dec 29, 2009 at 12:50:52PM +0000, Andy Green wrote:
> On 12/29/09 12:33, Somebody in the thread at some point said:

> >Compared to NAND removable media can be a win, yes, though with fast
> >JTAG having to rewrite the flash needn't be any slower so it's not quite
> >that clear cut.

> Right, but if you can eliminate JTAG in the flow, especially at the
> factory, that is a major simplification.

An awful lot of production setups are capable of handling JTAG already
so don't have much of an issue here.  Certainly as a developer my
preference here would be for fast JTAG over SD card - it's at least as
fast, there's only one system involved and it works just the same no
matter what the state of the system is.

> Again all I can say is it's specialized case.  For normal dev work
> SD is either painless or hugely advantageous.

Assuming the hardware can cope with it, and there's component cost,
board area and mechanical concerns to address before it gets designed
in.  There are a lot of systems where it would be useful but there's
drawbacks you have to bear in mind when pushing it.

Ethernet / SD Boot

[Andy Green]

On 12/30/09 13:16, Somebody in the thread at some point said:

Hi -

>> Right, but if you can eliminate JTAG in the flow, especially at the
>> factory, that is a major simplification.
>
> An awful lot of production setups are capable of handling JTAG already
> so don't have much of an issue here.  Certainly as a developer my
> preference here would be for fast JTAG over SD card - it's at least as
> fast, there's only one system involved and it works just the same no
> matter what the state of the system is.

If you have JTAG gear that works reliably (I never had much luck with 
OpenOCD, I only ever got it to work with the OM debug board with one 
specific build of a library, not the latest one either).  I have a Wind 
River JTAG dongle here that only works on Fedora 6 or something, it's 
not really workable here or at a factory to set up VMs to please this 
piece of test gear.  I guess you have better luck depending on JTAG than 
I have, it certainly creates a burden to use it in my experience.

Actually with SD basis you simply remove all other mass NV devices from 
your board.  SD cards can be duplicated out of the factory so there's no 
real comparison needed between raw NAND bringup over JTAG and SD Card basis.

The only piece of "board bringup" needed is to allocate and write board 
identity data somewhere like MACs if you want that to live on the board 
and not the SD, and that can be done from the test SD image acquiring 
the data over the network.

Customers don't have JTAG gear either so those arrangements and that 
work are specific to the production action.  However customers do have 
SD readers and that can be part of a debricking scheme that does not 
involve product return.

>> Again all I can say is it's specialized case.  For normal dev work
>> SD is either painless or hugely advantageous.
>
> Assuming the hardware can cope with it, and there's component cost,
> board area and mechanical concerns to address before it gets designed
> in.  There are a lot of systems where it would be useful but there's
> drawbacks you have to bear in mind when pushing it.

I don't really regard those as drawbacks.

Component cost is just the connector and four pullups, I guess it's $0.70.

Yeah you need to make the uSD connector accessible if you remove a 
cover, that's generally not hard.

CPUs in the class where it is useful (ARM11+) all tend to have one or 
more SD peripheral unit on the CPU already, so that's for free.

With true SD boot, cost of the SD card itself is balanced by being able 
to remove all

  - NAND

  - NOR

  - JTAG connector + passives

from the board, and all production and test actions around those now 
they are gone.

Space for the uSD socket on one side of the board or another is less 
than typical footprint of NAND and JTAG connector that used to be on the 
board.

I think you find SD basis is normally for free or cheaper.

There are a bunch of horrors around working with raw NAND that also 
disappear now your access to NAND is "cooked" by the controller on the 
SD Card.  You no longer need to deal with ECC, BBT, jffs2 and the like 
but can use normal filesystems directly.

So I think the issue is more the tendency to not bear the drawbacks of 
existing schemes and bootloaders in mind when considering different 
workflows, since dealing with the issues from those are what people are 
used to.

-Andy

Ethernet / SD Boot

[Mark Brown]

On Wed, Dec 30, 2009 at 01:37:34PM +0000, Andy Green wrote:

> If you have JTAG gear that works reliably (I never had much luck
> with OpenOCD, I only ever got it to work with the OM debug board
> with one specific build of a library, not the latest one either).  I
> have a Wind River JTAG dongle here that only works on Fedora 6 or
> something, it's not really workable here or at a factory to set up
> VMs to please this piece of test gear.  I guess you have better luck
> depending on JTAG than I have, it certainly creates a burden to use
> it in my experience.

The one I've had the most positive experience with was the BDI2000 - no
host software required, it's controlled over the network via telnet (and
possibly other things now) for non-debugging activity and provides a GDB
stub for debugging.  Like everything else there are plenty of rubbish
products out there.

> >Assuming the hardware can cope with it, and there's component cost,
> >board area and mechanical concerns to address before it gets designed
> >in.  There are a lot of systems where it would be useful but there's
> >drawbacks you have to bear in mind when pushing it.

> I don't really regard those as drawbacks.

> Component cost is just the connector and four pullups, I guess it's $0.70.

That 70 cents does rather add up once you start to ship in volume, even
a fraction of that would.

> Yeah you need to make the uSD connector accessible if you remove a
> cover, that's generally not hard.

In some form factors.  In others it's very problematic.

> CPUs in the class where it is useful (ARM11+) all tend to have one
> or more SD peripheral unit on the CPU already, so that's for free.

Assuming you've not used all the pins for something else, which is again
a problem for some classes of device.

Ethernet / SD Boot

[Andy Green]

On 12/31/09 12:27, Somebody in the thread at some point said:

Hi -

> The one I've had the most positive experience with was the BDI2000 - no
> host software required, it's controlled over the network via telnet (and
> possibly other things now) for non-debugging activity and provides a GDB
> stub for debugging.  Like everything else there are plenty of rubbish
> products out there.

Sounds pretty good, thanks for the tip.

>>> Assuming the hardware can cope with it, and there's component cost,
>>> board area and mechanical concerns to address before it gets designed
>>> in.  There are a lot of systems where it would be useful but there's
>>> drawbacks you have to bear in mind when pushing it.
>
>> I don't really regard those as drawbacks.
>
>> Component cost is just the connector and four pullups, I guess it's $0.70.
>
> That 70 cents does rather add up once you start to ship in volume, even
> a fraction of that would.

You did notice that you get to save money back from removing NAND and 
JTAG connectors?

Due to the other advantages of SD Boot basis you can also expect to get 
your product to market quicker, spend less time in the factory, and have 
less returns all of which save money.

So I wouldn't let this notional $0.70 put you off.

>> Yeah you need to make the uSD connector accessible if you remove a
>> cover, that's generally not hard.
>
> In some form factors.  In others it's very problematic.

Dunno how problematic it can be actually, unless you dunk your board in 
epoxy.  uSD connectors are very small footprint and low profile since 
they're widely used in consumer equipment like phones.  Since you only 
need access to it in a crisis due to bricking it, it's OK if you have to 
remove a cover or whatever.  Space shouldn't be an issue due to what it 
allows you to throw out from the board.  I'm sure there are special 
cases but I maintain "generally, that's not hard".

>> CPUs in the class where it is useful (ARM11+) all tend to have one
>> or more SD peripheral unit on the CPU already, so that's for free.
>
> Assuming you've not used all the pins for something else, which is again
> a problem for some classes of device.

Yeah it's true, if the mux arrangements conflict with other peripheral 
units you must have something's got to give.

But many chips have 2 or 3 SD units, mux arrangements that bring the 
same things to different pins.  In the case of SD interface, it's a 
given in context of phones where a lot of these chips go, so you tend 
not to find other IO on those balls that is critical and does not also 
appear on other ball muxes.  (It's only 6 balls anyway, it's not like a 
memory bus).  It seems you can typically expect you're going to be able 
to use SD unless you find a specific conflict because the chips are 
deliberately designed to enable SD usage in typical design.

-Andy

Ethernet / SD Boot

[Ben Dooks]

Andy Green wrote:
> On 12/31/09 12:27, Somebody in the thread at some point said:
> 
> Hi -
> 
>> The one I've had the most positive experience with was the BDI2000 - no
>> host software required, it's controlled over the network via telnet (and
>> possibly other things now) for non-debugging activity and provides a GDB
>> stub for debugging.  Like everything else there are plenty of rubbish
>> products out there.
> 
> Sounds pretty good, thanks for the tip.

I pine for the BDI2000 I was lent...

Of course, with bed of nails type systems it is probably possible to
allow the system to boot off what you like, NAND is easily ~18
connections to the board, SD is of course less.

>>>> Assuming the hardware can cope with it, and there's component cost,
>>>> board area and mechanical concerns to address before it gets designed
>>>> in.  There are a lot of systems where it would be useful but there's
>>>> drawbacks you have to bear in mind when pushing it.
>>
>>> I don't really regard those as drawbacks.
>>
>>> Component cost is just the connector and four pullups, I guess it's
>>> $0.70.
>>
>> That 70 cents does rather add up once you start to ship in volume, even
>> a fraction of that would.
> 
> You did notice that you get to save money back from removing NAND and
> JTAG connectors?

A lot of times these are done as 'bed of nails' pads on the PCB so
hardly a bank breaker... Also, see PoP argument below.

> Due to the other advantages of SD Boot basis you can also expect to get
> your product to market quicker, spend less time in the factory, and have
> less returns all of which save money.

And as shown by the nook, make it really easy for your users to root the
thing without trying. You've been spoiled a bit by having an open design
to play with, not everyone likes their users messing with the hardware
once it has left the factory.

Also as another cost item, a lot of the newer cpus with PoP style memory
have some form of NAND/OneNAND built into the system already so adding
an extra SD card is often seen as a waste of extant storage.

> So I wouldn't let this notional $0.70 put you off.
> 
>>> Yeah you need to make the uSD connector accessible if you remove a
>>> cover, that's generally not hard.
>>
>> In some form factors.  In others it's very problematic.
> 
> Dunno how problematic it can be actually, unless you dunk your board in
> epoxy.  uSD connectors are very small footprint and low profile since
> they're widely used in consumer equipment like phones.  Since you only
> need access to it in a crisis due to bricking it, it's OK if you have to
> remove a cover or whatever.  Space shouldn't be an issue due to what it
> allows you to throw out from the board.  I'm sure there are special
> cases but I maintain "generally, that's not hard".

A lot of the time, the bigger manufacturers have enough money that a
commercial JTAG system makes sense, as not only can it program the board
but there are also connectivity checking systems that allow you to do
signal connectiivty testing and other such wonderfullnesses for you.

Smaller vendors tend to cobble together something using open-source
tools, etc.

>>> CPUs in the class where it is useful (ARM11+) all tend to have one
>>> or more SD peripheral unit on the CPU already, so that's for free.
>>
>> Assuming you've not used all the pins for something else, which is again
>> a problem for some classes of device.
> 
> Yeah it's true, if the mux arrangements conflict with other peripheral
> units you must have something's got to give.
> 
> But many chips have 2 or 3 SD units, mux arrangements that bring the
> same things to different pins.  In the case of SD interface, it's a
> given in context of phones where a lot of these chips go, so you tend
> not to find other IO on those balls that is critical and does not also
> appear on other ball muxes.  (It's only 6 balls anyway, it's not like a
> memory bus).  It seems you can typically expect you're going to be able
> to use SD unless you find a specific conflict because the chips are
> deliberately designed to enable SD usage in typical design.

and of course, a lot of modern mobile devices need SD for talking to
other devices, such as WLAN chips, external storage cards, etc. It can
soon take a toll on what is available.

Ethernet / SD Boot

[Andy Green]

On 01/04/10 16:26, Somebody in the thread at some point said:

Hi -

> Of course, with bed of nails type systems it is probably possible to
> allow the system to boot off what you like, NAND is easily ~18
> connections to the board, SD is of course less.

Yeah... all kinds of complex board bringup schemes are possible... 
but... are they wise?  It's special for the factory and special for the 
pcb revision.  It's extra cost and management.

For example Openmoko went your NAND bringup route and it was basically 
an intricate, complex Hell that few people understood and fewer, like 
zero, actually wanted to go to the factory with.  OpenOCD basis turned 
it into a lottery if even internal devs could work with it.

For smaller runs test jigs are popular, they have the same problem of 
requiring developer and design effort.

Often when I have this conversation, it mainly runs into the other guy 
having heard / decided that "real men" must have jigs / jtag / u-boot or 
whatever it is.  But actually, it can be a huge amount of dead weight 
around small to medium run development effort and it can be entirely 
optional to tie those stones around your neck.

Maybe you like sorting that out but I really prefer the SD Boot scheme 
where there is no factory bringup arrangement at all except the shipping 
rootfs with some bash scripts turning a board from the oven into a 
shipping device.

>>> That 70 cents does rather add up once you start to ship in volume, even
>>> a fraction of that would.
>>
>> You did notice that you get to save money back from removing NAND and
>> JTAG connectors?
>
> A lot of times these are done as 'bed of nails' pads on the PCB so
> hardly a bank breaker... Also, see PoP argument below.

You're referring to NAND programming here I assume.

Something has to be on the other end of those spring probes / jig and it 
has to be prepared / maintained / tested.  The burden does not end with 
the fact you made contact with some probes.  It's special for the 
factory and cannot be repeated by the user (or even the devs without 
grief).  If you make another run two years later and the guy who did the 
magic left, maybe the processing steps to turn an image into SVF is lost 
and suddenly the visit to the factory is a risk.

A better argument for that I have heard would be, "well, I have to 100% 
test connectivity on all my external ports, don't I, spring probes will 
solve it".  But this argument is also broken, you are not testing 
connector function that way.

If it's doable on the design, the optimal test strategy is often 
loopback plugs and selftest, although how far you can get with that 
depends on exactly what your device exports on the connectors.

>> Due to the other advantages of SD Boot basis you can also expect to get
>> your product to market quicker, spend less time in the factory, and have
>> less returns all of which save money.
>
> And as shown by the nook, make it really easy for your users to root the
> thing without trying. You've been spoiled a bit by having an open design
> to play with, not everyone likes their users messing with the hardware
> once it has left the factory.

No, that does not have to follow.

iMX31 has hard crypto chain of trust all the way from ROM, through SD 
bootloader being signed, which can then validate signatures of kernel 
and rootfs.  If you want it locked up and do true SD boot you can do it.

Further you are mistaken if you think having a NAND on a board is any 
impediment to getting / changing its contents to someone who wants to do 
that.  It makes it harder to change the contents without swapping the 
chip, but that's a double edge sword in debrickability terms when you 
only targeted a factory with a jig to be able to do that.

And modchip folks will tell you that once you have the rootfs, you stand 
a decent chance of finding something that may let you circumvent 
security and reflash it without NAND lifting.

You need a bit more of a story than you just put the same stuff in a 
NAND on your board and it's Fort Knox now.

> Also as another cost item, a lot of the newer cpus with PoP style memory
> have some form of NAND/OneNAND built into the system already so adding
> an extra SD card is often seen as a waste of extant storage.

Some do PoP and some don't; the mix of assets in the PoP is soft if you 
are a large vendor, so it's possible to just get DDR in there.  DDR is 
the thing where PoP actually delivers something, since it removed the 
board layout and verification burden.

>> So I wouldn't let this notional $0.70 put you off.
>>
>>>> Yeah you need to make the uSD connector accessible if you remove a
>>>> cover, that's generally not hard.
>>>
>>> In some form factors.  In others it's very problematic.
>>
>> Dunno how problematic it can be actually, unless you dunk your board in
>> epoxy.  uSD connectors are very small footprint and low profile since
>> they're widely used in consumer equipment like phones.  Since you only
>> need access to it in a crisis due to bricking it, it's OK if you have to
>> remove a cover or whatever.  Space shouldn't be an issue due to what it
>> allows you to throw out from the board.  I'm sure there are special
>> cases but I maintain "generally, that's not hard".
>
> A lot of the time, the bigger manufacturers have enough money that a
> commercial JTAG system makes sense, as not only can it program the board
> but there are also connectivity checking systems that allow you to do
> signal connectiivty testing and other such wonderfullnesses for you.

Yeah.  But many (most?) folks working on this stuff are trying to bring 
complex products to market with insufficient resources and pressed for 
time.  There isn't any internal test team.  What there is mainly is the 
choices that the developer will make as to how to come at production.

By smart choices about how to

  - boot

  - create the rootfs

  - test the device

the developers can eliminate several kinds of downstream steps that are 
currently considered unavoidable and get the optimal situation in 
previously dangerous areas like board bringup and rootfs maintainence.

>> But many chips have 2 or 3 SD units, mux arrangements that bring the
>> same things to different pins.  In the case of SD interface, it's a
>> given in context of phones where a lot of these chips go, so you tend
>> not to find other IO on those balls that is critical and does not also
>> appear on other ball muxes.  (It's only 6 balls anyway, it's not like a
>> memory bus).  It seems you can typically expect you're going to be able
>> to use SD unless you find a specific conflict because the chips are
>> deliberately designed to enable SD usage in typical design.
>
> and of course, a lot of modern mobile devices need SD for talking to
> other devices, such as WLAN chips, external storage cards, etc. It can
> soon take a toll on what is available.

Sure, if you need more SD units than the chip has, something's gonna 
have to give for sure.  But I look around and most things I see don't 
have 3 SDs used up.

The last few objections have basically been, "what if I can't use SD 
Boot for some reason?  Where is your God NOW???"

If you actually can't use it because you dip your board in epoxy; you 
need all the SD ports for something else; your CPU doesn't have SD boot; 
your customer thinks a NAND on a PCB is secure... then don't bother 
considering what you can't use and do something else :-)

Otherwise, you should study carefully the advantages you can get from 
true SD Boot in simplifying the hardware, software engineering and 
production / test / debrick.  It has pretty revolutionary benefits.

-Andy

Ethernet / SD Boot

[Russell King - ARM Linux]

On Mon, Jan 04, 2010 at 05:28:37PM +0000, Andy Green wrote:
> Otherwise, you should study carefully the advantages you can get from  
> true SD Boot in simplifying the hardware, software engineering and  
> production / test / debrick.  It has pretty revolutionary benefits.

Look, it's very obvious from every one of your replies that you think
the sun shines out of your SD cards, it's superiour to whatever else
people are using, and that everyone should use it.

You have an answer for every argument to the contary.

Well, fine, you're entitled to that opinion.

Now can we stop discussing this and move on please.

Ethernet / SD Boot

[Andy Green]

On 01/04/10 17:40, Somebody in the thread at some point said:
> On Mon, Jan 04, 2010 at 05:28:37PM +0000, Andy Green wrote:
>> Otherwise, you should study carefully the advantages you can get from
>> true SD Boot in simplifying the hardware, software engineering and
>> production / test / debrick.  It has pretty revolutionary benefits.
>
> Look, it's very obvious from every one of your replies that you think
> the sun shines out of your SD cards, it's superiour to whatever else
> people are using, and that everyone should use it.
>
> You have an answer for every argument to the contrary.

Yeah, that's normally considered a good sign for a thesis.

> Well, fine, you're entitled to that opinion.
>
> Now can we stop discussing this and move on please.

Sure.

However if anyone else is interested in help setting up with it, feel 
free to contact me direct.

-Andy

Ethernet / SD Boot

[Marc Zyngier]

On Mon, 04 Jan 2010 18:09:26 +0000
Andy Green <andy@warmcat.com> wrote:

Hi Andy,

> However if anyone else is interested in help setting up with it, feel 
> free to contact me direct.

I was trying to access your Qi tree
(http://git.warmcat.com/cgi-bin/cgit/qi/), without any success (it
times out after a while...).

Is this intentional?

Thanks,

	M.
-- 
Fast. Cheap. Reliable. Pick two.

Ethernet / SD Boot

[Jamie Lokier]

Fwiw:

1. 100% of CPUs I've used don't have SD boot.
2. I don't want to fiddle about with removing and replacing an SD card
   every time I change the kernel / bootloader.
   Give me a cable and computer!
3. It's good to have a debricking strategy, but even better if it's the
   same as or similar to the regular development strategy - better
   test coverage, less knowledge to get lost.
4. Thus hardware USB boot is surely best, and TFTP the next best :-)
5. JTAG board connectivity is sometimes done with Perl and parallel ports
   too; it's not reserved only for expensive dev packages.

-- Jamie