[U-Boot] incremental environment updating

[U-Boot] incremental environment updating

[Mike Frysinger]

currently the env code will erase the entire env storage before writing back 
out the current env, even if the env storage has enough empty space to store 
the current env.  for example, if CONFIG_ENV_SIZE is declared as 0x2000 but 
the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and 
then the ~0x300 gets written out.  seems like we can get a pretty good return 
for fairly low effort if we appended env updates rather than erasing/writing 
every time ?  it'd certainly be faster.  while systems with a dedicated sector 
this isnt so bad, but for people who have to embed the env in the middle of a 
large sector, this would be much faster most of the time.

has there been previous discussion along these lines that i havent seen ?
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090413/88540c1f/attachment.pgp 

[U-Boot] incremental environment updating

[Wolfgang Denk]

Dear Mike,

In message <200904130609.08060.vapier@gentoo.org> you wrote:
>
> currently the env code will erase the entire env storage before writing back 
> out the current env, even if the env storage has enough empty space to store 
> the current env.  for example, if CONFIG_ENV_SIZE is declared as 0x2000 but 
> the current env only takes up ~0x300 bytes, the whole 0x2000 is erased and
> then the ~0x300 gets written out.  seems like we can get a pretty good return 
> for fairly low effort if we appended env updates rather than erasing/writing 
> every time ?  it'd certainly be faster.  while systems with a dedicated sector 
> this isnt so bad, but for people who have to embed the env in the middle of a 
> large sector, this would be much faster most of the time.
>
> has there been previous discussion along these lines that i havent seen ?

This hasn't been discussed before. Interesting idea. However, I fail
to see how this could be implemented without changing the environment
format?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
I express preference for a chronological  sequence  of  events  which
precludes a violence.   - Terry Pratchett, _The Dark Side of the Sun_

[U-Boot] incremental environment updating

[Mike Frysinger]

On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
> In message Mike wrote:
> > currently the env code will erase the entire env storage before writing
> > back out the current env, even if the env storage has enough empty space
> > to store the current env.  for example, if CONFIG_ENV_SIZE is declared as
> > 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000
> > is erased and then the ~0x300 gets written out.  seems like we can get a
> > pretty good return for fairly low effort if we appended env updates
> > rather than erasing/writing every time ?  it'd certainly be faster. 
> > while systems with a dedicated sector this isnt so bad, but for people
> > who have to embed the env in the middle of a large sector, this would be
> > much faster most of the time.
> >
> > has there been previous discussion along these lines that i havent seen ?
>
> This hasn't been discussed before. Interesting idea. However, I fail
> to see how this could be implemented without changing the environment
> format?

that depends on how you want the compatibility to go.  being able to read old 
environments by newer u-boots is reasonable, but i dont think having old u-
boots read newer environments makes realistic sense ?

in terms of actual changes, i had a couple of ideas ... the current env format 
is: <crc><env><NUL>[undefined].  so if we logically extend the format where 
[undefined] is <crc><env><NUL>[...], then all existing env storage would be 
automatically imported.  considering most env storage out there uses a bit 
value of "0" to mean programmed and "1' to mean unprogrammed, it should be 
pretty easy to quickly detect where the appended envs stop.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090413/3a87fa60/attachment.pgp 

[U-Boot] incremental environment updating

[Jerry Van Baren]

Mike Frysinger wrote:
> On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
>> In message Mike wrote:
>>> currently the env code will erase the entire env storage before writing
>>> back out the current env, even if the env storage has enough empty space
>>> to store the current env.  for example, if CONFIG_ENV_SIZE is declared as
>>> 0x2000 but the current env only takes up ~0x300 bytes, the whole 0x2000
>>> is erased and then the ~0x300 gets written out.  seems like we can get a
>>> pretty good return for fairly low effort if we appended env updates
>>> rather than erasing/writing every time ?  it'd certainly be faster. 
>>> while systems with a dedicated sector this isnt so bad, but for people
>>> who have to embed the env in the middle of a large sector, this would be
>>> much faster most of the time.
>>>
>>> has there been previous discussion along these lines that i havent seen ?
>> This hasn't been discussed before. Interesting idea. However, I fail
>> to see how this could be implemented without changing the environment
>> format?
> 
> that depends on how you want the compatibility to go.  being able to read old 
> environments by newer u-boots is reasonable, but i dont think having old u-
> boots read newer environments makes realistic sense ?
> 
> in terms of actual changes, i had a couple of ideas ... the current env format 
> is: <crc><env><NUL>[undefined].  so if we logically extend the format where 
> [undefined] is <crc><env><NUL>[...], then all existing env storage would be 
> automatically imported.  considering most env storage out there uses a bit 
> value of "0" to mean programmed and "1' to mean unprogrammed, it should be 
> pretty easy to quickly detect where the appended envs stop.
> -mike

Hi Mike,

Another concept is to append a complete new env on every write.  This is 
less efficient than just a delta, but has a substantial advantage in 
that it gives the user a way to erase env variables as well as change 
them.  I suspect it would also be simpler to implement.

Since flash can change 1s to 0s, but cannot change them back, you could 
redefine the env storage to be
   <next><crc><env><NUL>[undefined]
(note the added <next> at the start).  By definition, the valid env's 
<next> == 0xFFFFFFFF.  To write a new env, simply write the offset of 
[undefined] into the <next> location and then write a new env lump.
   <next><crc><env><NUL> <next><crc><env><NUL>[undefined]
    ^^^^ offset of -----> ^^^^ 0xFFFFFFFF

To find the valid env, you simply start at the start of the env area and 
keep adding <next> to the env start address until <next> == 0xFFFFFFFF. 
  Add four and your env address is correct.  No other change needed to 
the env reading code.

Either way, we would need a new command, say "compactenv" (kind of a 
long command :-(, maybe there is a shorter German word ;-).  This would 
erase the current env sector and rewrite it with one env, resetting the 
clock.  This could also be done automatically if there is no room left 
in the env storage area (both options would probably be good to implement).

Another note: this would make it safer to append the env to u-boot 
rather than keeping it in a separate sector, except when the env storage 
area needs to be cleared.  I don't know if that is worth bragging about, 
however ("Our New Improved[tm] Env storage handling is 99 times safer: 
it now only risks bricking your board one time in a hundred!").

Best regards,
gvb

[U-Boot] incremental environment updating

[Wolfgang Denk]

Dear Jerry Van Baren,

In message <49E32C3C.8090404@ge.com> you wrote:
>
> Since flash can change 1s to 0s, but cannot change them back, you could 
> redefine the env storage to be
>    <next><crc><env><NUL>[undefined]
> (note the added <next> at the start).  By definition, the valid env's 
> <next> == 0xFFFFFFFF.  To write a new env, simply write the offset of 
> [undefined] into the <next> location and then write a new env lump.

I see problems with this:

- it's incompatible with the current format, i. e. detection of
  existing old formats needs a special case which is something I
  always consider ugly.

- it works only on NOR flash, not on any media that can be accessed
  only in blocks

- handling of redundant environment becomes a lot more complicated


BTW: what was the exact problem we were trying to solve? Saving the
time it takes to erase one flash sector?

If we have to pay  for  this  with  introducing  a  new,  incomptible
environment format, and probably having to handle NOR flash and other
storage media differently, I doubt if this is worth the effort.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Imagination is more important than knowledge.      -- Albert Einstein

[U-Boot] incremental environment updating

[Mike Frysinger]

On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
> Mike Frysinger wrote:
> > On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
> >> In message Mike wrote:
> >>> currently the env code will erase the entire env storage before writing
> >>> back out the current env, even if the env storage has enough empty
> >>> space to store the current env.  for example, if CONFIG_ENV_SIZE is
> >>> declared as 0x2000 but the current env only takes up ~0x300 bytes, the
> >>> whole 0x2000 is erased and then the ~0x300 gets written out.  seems
> >>> like we can get a pretty good return for fairly low effort if we
> >>> appended env updates rather than erasing/writing every time ?  it'd
> >>> certainly be faster. while systems with a dedicated sector this isnt so
> >>> bad, but for people who have to embed the env in the middle of a large
> >>> sector, this would be much faster most of the time.
> >>>
> >>> has there been previous discussion along these lines that i havent seen
> >>> ?
> >>
> >> This hasn't been discussed before. Interesting idea. However, I fail
> >> to see how this could be implemented without changing the environment
> >> format?
> >
> > that depends on how you want the compatibility to go.  being able to read
> > old environments by newer u-boots is reasonable, but i dont think having
> > old u- boots read newer environments makes realistic sense ?
> >
> > in terms of actual changes, i had a couple of ideas ... the current env
> > format is: <crc><env><NUL>[undefined].  so if we logically extend the
> > format where [undefined] is <crc><env><NUL>[...], then all existing env
> > storage would be automatically imported.  considering most env storage
> > out there uses a bit value of "0" to mean programmed and "1' to mean
> > unprogrammed, it should be pretty easy to quickly detect where the
> > appended envs stop.
>
> Another concept is to append a complete new env on every write.  This is
> less efficient than just a delta, but has a substantial advantage in
> that it gives the user a way to erase env variables as well as change
> them.  I suspect it would also be simpler to implement.

this is actually what i was talking about.  doing a delta would require the 
env code to compare the old env and "whiteout" variables that were set but are 
now unset, and it would take longer to boot up as the env code would have to 
rebuild using the deltas.

> Since flash can change 1s to 0s, but cannot change them back, you could
> redefine the env storage to be
>    <next><crc><env><NUL>[undefined]
> (note the added <next> at the start).  By definition, the valid env's
> <next> == 0xFFFFFFFF.  To write a new env, simply write the offset of
> [undefined] into the <next> location and then write a new env lump.
>    <next><crc><env><NUL> <next><crc><env><NUL>[undefined]
>     ^^^^ offset of -----> ^^^^ 0xFFFFFFFF

using a crc of 0x00000000 accomplishes the same thing and retains env format

> Either way, we would need a new command, say "compactenv" (kind of a
> long command :-(, maybe there is a shorter German word ;-).  This would
> erase the current env sector and rewrite it with one env, resetting the
> clock.  This could also be done automatically if there is no room left
> in the env storage area (both options would probably be good to implement).

i didnt intend for any external behavior change.  the env_*.c would take care 
of the appending and/or reset step.  every thing else would continue to 
operate the same way.  along those lines, i dont really see the need for 
"compactenv" beyond a helper command when debugging implementation.  there 
would be no need to manually run this step.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090413/d48e9775/attachment.pgp 

[U-Boot] incremental environment updating

[Jerry Van Baren]

Wolfgang Denk wrote:
> Dear Jerry Van Baren,
> 
> In message <49E32C3C.8090404@ge.com> you wrote:
>> Since flash can change 1s to 0s, but cannot change them back, you could 
>> redefine the env storage to be
>>    <next><crc><env><NUL>[undefined]
>> (note the added <next> at the start).  By definition, the valid env's 
>> <next> == 0xFFFFFFFF.  To write a new env, simply write the offset of 
>> [undefined] into the <next> location and then write a new env lump.
> 
> I see problems with this:
> 
> - it's incompatible with the current format, i. e. detection of
>   existing old formats needs a special case which is something I
>   always consider ugly.

True, although that is an issue only for upgrades to boards that have an 
existing env and that you want to use the new env method on.  I would 
expect the special casing would not be excessively ugly and may not be 
necessary, depending on how important backward compatibility (board 
upgrading) is.

> - it works only on NOR flash, not on any media that can be accessed
>   only in blocks

Not true: the offset merely needs to be aligned to the next block in 
that case:
   <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined]
   ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF

All flash that I can recall seeing allows at least a limited number of 
rewrites to a block (the limited number getting smaller on successive 
generations), so two writes to the env start block (once to write the 
new env, a second time to rewrite the 0xFFFFFFFF to point to the 
superceeding env).  If the flash didn't support two writes, that would 
be a show-stopper.  Due to how people use flash, I don't think the 
number will ever go to one write (IIRC, 4 is typical of current 
generation NAND).

> - handling of redundant environment becomes a lot more complicated

I don't think so: it would be handled the same way as the primary env in 
a separate sector as is done currently.  If you step through all of the 
primary env lumps and the last one has an invalid CRC, you fall back to 
the backup env and repeat the search.

> BTW: what was the exact problem we were trying to solve? Saving the
> time it takes to erase one flash sector?
> 
> If we have to pay  for  this  with  introducing  a  new, incompatible
> environment format, and probably having to handle NOR flash and other
> storage media differently, I doubt if this is worth the effort.

That could very well be true.

> Best regards,
> 
> Wolfgang Denk

Ditto,
gvb

[U-Boot] incremental environment updating

[Wolfgang Denk]

Dear Mike,

In message <200904130834.35359.vapier@gentoo.org> you wrote:
>
> > Since flash can change 1s to 0s, but cannot change them back, you could
> > redefine the env storage to be
> >    <next><crc><env><NUL>[undefined]
> > (note the added <next> at the start).  By definition, the valid env's
> > <next> =3D=3D 0xFFFFFFFF.  To write a new env, simply write the offset of
> > [undefined] into the <next> location and then write a new env lump.
> >    <next><crc><env><NUL> <next><crc><env><NUL>[undefined]
> >     ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
> 
> using a crc of 0x00000000 accomplishes the same thing and retains env format

No, this is not the same. With the offset, you can jump  directly  to
the  next  block;  with  just a "valid" flag you have to scan invalid
copy sequentially to find  the  end  -  especially  on  EEPROM  based
storage this is painfully slow (see
http://www.denx.de/wiki/DULG/AN2004_11_BootTimeOptimization).


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
It would seem that evil retreats when forcibly confronted
	-- Yarnek of Excalbia, "The Savage Curtain", stardate 5906.5

[U-Boot] incremental environment updating

[Wolfgang Denk]

Dear Jerry Van Baren,

In message <49E335BB.4050609@ge.com> you wrote:
>
> > - it works only on NOR flash, not on any media that can be accessed
> >   only in blocks
> 
> Not true: the offset merely needs to be aligned to the next block in 
> that case:
>    <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined]
>    ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF
> 
> All flash that I can recall seeing allows at least a limited number of 
> rewrites to a block (the limited number getting smaller on successive 
> generations), so two writes to the env start block (once to write the 
> new env, a second time to rewrite the 0xFFFFFFFF to point to the 
> superceeding env).  If the flash didn't support two writes, that would 
> be a show-stopper.  Due to how people use flash, I don't think the 
> number will ever go to one write (IIRC, 4 is typical of current 
> generation NAND).

So what do we save, then? Instead of a single write we now need
actually two...

> > BTW: what was the exact problem we were trying to solve? Saving the
> > time it takes to erase one flash sector?
> > 
> > If we have to pay  for  this  with  introducing  a  new, incompatible
> > environment format, and probably having to handle NOR flash and other
> > storage media differently, I doubt if this is worth the effort.
> 
> That could very well be true.

;-)

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
For every complex problem, there is a solution that is simple,  neat,
and wrong.                                           -- H. L. Mencken

[U-Boot] incremental environment updating

[Jerry Van Baren]

Mike Frysinger wrote:
> On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
>> Mike Frysinger wrote:
>>> On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
>>>> In message Mike wrote:
>>>>> currently the env code will erase the entire env storage before writing
>>>>> back out the current env, even if the env storage has enough empty
>>>>> space to store the current env.  for example, if CONFIG_ENV_SIZE is
>>>>> declared as 0x2000 but the current env only takes up ~0x300 bytes, the
>>>>> whole 0x2000 is erased and then the ~0x300 gets written out.  seems
>>>>> like we can get a pretty good return for fairly low effort if we
>>>>> appended env updates rather than erasing/writing every time ?  it'd
>>>>> certainly be faster. while systems with a dedicated sector this isnt so
>>>>> bad, but for people who have to embed the env in the middle of a large
>>>>> sector, this would be much faster most of the time.

Hmmm, rereading this, if the issue is simply the time to erase a sector, 
I don't think this is that big of a deal.  It takes ~2 seconds? to erase 
a sector and env variables are rarely changed in my experience.  Does 
your mileage vary?

>>>>> has there been previous discussion along these lines that i havent seen
>>>>> ?
>>>> This hasn't been discussed before. Interesting idea. However, I fail
>>>> to see how this could be implemented without changing the environment
>>>> format?
>>> that depends on how you want the compatibility to go.  being able to read
>>> old environments by newer u-boots is reasonable, but i dont think having
>>> old u- boots read newer environments makes realistic sense ?
>>>
>>> in terms of actual changes, i had a couple of ideas ... the current env
>>> format is: <crc><env><NUL>[undefined].  so if we logically extend the
>>> format where [undefined] is <crc><env><NUL>[...], then all existing env
>>> storage would be automatically imported.  considering most env storage
>>> out there uses a bit value of "0" to mean programmed and "1' to mean
>>> unprogrammed, it should be pretty easy to quickly detect where the
>>> appended envs stop.
>> Another concept is to append a complete new env on every write.  This is
>> less efficient than just a delta, but has a substantial advantage in
>> that it gives the user a way to erase env variables as well as change
>> them.  I suspect it would also be simpler to implement.
> 
> this is actually what i was talking about.  doing a delta would require the 
> env code to compare the old env and "whiteout" variables that were set but are 
> now unset, and it would take longer to boot up as the env code would have to 
> rebuild using the deltas.

I'm confused, are we actually proposing the same thing?  When you say 
"delta" and "whiteout" I picture additional definitions appended as a 
delta env lump to the original env lump, not a whole new env lump.

My counterproposal eliminates the delta determination software and 
totally eliminates having to invent a new way to "whiteout" to indicate 
removed variables (IIRC, it is valid to set a variable with no value).

Further, once you find "the" env lump, the env processing is 100% the 
same so the implementation of my counterproposal simply wraps that 
existing env handling code with a "find the latest lump" loop.

>> Since flash can change 1s to 0s, but cannot change them back, you could
>> redefine the env storage to be
>>    <next><crc><env><NUL>[undefined]
>> (note the added <next> at the start).  By definition, the valid env's
>> <next> == 0xFFFFFFFF.  To write a new env, simply write the offset of
>> [undefined] into the <next> location and then write a new env lump.
>>    <next><crc><env><NUL> <next><crc><env><NUL>[undefined]
>>     ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
> 
> using a crc of 0x00000000 accomplishes the same thing and retains env format

Three problems:
1) 0x00000000 can be a valid CRC.  You either have a risk of failure 
(granted, very small) or you have to special case that instance.  Note 
that, if you don't handle the 0x00000000 case properly, you will 
probably brick the board.

2) Handling a redundant env may be more difficult.  OK, that probably 
isn't too bad, assuming you can handle the 0x00000000 CRC value properly 
since a bad env would have a non-zero invalid CRC.

3) My <next> offset is much faster since you don't have to step through 
the old env to find where the next one starts.  It also handles 
Wolfgang's point with NAND flashes needing to be aligned to blocks. 
(What do you do with a non-zero bad CRC?  Fall back to the redundant env?)

>> Either way, we would need a new command, say "compactenv" (kind of a
>> long command :-(, maybe there is a shorter German word ;-).  This would
>> erase the current env sector and rewrite it with one env, resetting the
>> clock.  This could also be done automatically if there is no room left
>> in the env storage area (both options would probably be good to implement).
> 
> i didnt intend for any external behavior change.  the env_*.c would take care 
> of the appending and/or reset step.  every thing else would continue to 
> operate the same way.  along those lines, i dont really see the need for 
> "compactenv" beyond a helper command when debugging implementation.  there 
> would be no need to manually run this step.
> -mike

OK.

gvb

[U-Boot] incremental environment updating

[Mike Frysinger]

On Monday 13 April 2009 09:11:59 Jerry Van Baren wrote:
> Mike Frysinger wrote:
> > On Monday 13 April 2009 08:12:44 Jerry Van Baren wrote:
> >> Mike Frysinger wrote:
> >>> On Monday 13 April 2009 06:15:24 Wolfgang Denk wrote:
> >>>> In message Mike wrote:
> >>>>> currently the env code will erase the entire env storage before
> >>>>> writing back out the current env, even if the env storage has enough
> >>>>> empty space to store the current env.  for example, if
> >>>>> CONFIG_ENV_SIZE is declared as 0x2000 but the current env only takes
> >>>>> up ~0x300 bytes, the whole 0x2000 is erased and then the ~0x300 gets
> >>>>> written out.  seems like we can get a pretty good return for fairly
> >>>>> low effort if we appended env updates rather than erasing/writing
> >>>>> every time ?  it'd certainly be faster. while systems with a
> >>>>> dedicated sector this isnt so bad, but for people who have to embed
> >>>>> the env in the middle of a large sector, this would be much faster
> >>>>> most of the time.
>
> Hmmm, rereading this, if the issue is simply the time to erase a sector,
> I don't think this is that big of a deal.  It takes ~2 seconds? to erase
> a sector and env variables are rarely changed in my experience.  Does
> your mileage vary?

speed, reliability, and wear

> >>>>> has there been previous discussion along these lines that i havent
> >>>>> seen ?
> >>>>
> >>>> This hasn't been discussed before. Interesting idea. However, I fail
> >>>> to see how this could be implemented without changing the environment
> >>>> format?
> >>>
> >>> that depends on how you want the compatibility to go.  being able to
> >>> read old environments by newer u-boots is reasonable, but i dont think
> >>> having old u- boots read newer environments makes realistic sense ?
> >>>
> >>> in terms of actual changes, i had a couple of ideas ... the current env
> >>> format is: <crc><env><NUL>[undefined].  so if we logically extend the
> >>> format where [undefined] is <crc><env><NUL>[...], then all existing env
> >>> storage would be automatically imported.  considering most env storage
> >>> out there uses a bit value of "0" to mean programmed and "1' to mean
> >>> unprogrammed, it should be pretty easy to quickly detect where the
> >>> appended envs stop.
> >>
> >> Another concept is to append a complete new env on every write.  This is
> >> less efficient than just a delta, but has a substantial advantage in
> >> that it gives the user a way to erase env variables as well as change
> >> them.  I suspect it would also be simpler to implement.
> >
> > this is actually what i was talking about.  doing a delta would require
> > the env code to compare the old env and "whiteout" variables that were
> > set but are now unset, and it would take longer to boot up as the env
> > code would have to rebuild using the deltas.
>
> I'm confused, are we actually proposing the same thing?  When you say
> "delta" and "whiteout" I picture additional definitions appended as a
> delta env lump to the original env lump, not a whole new env lump.
>
> My counterproposal eliminates the delta determination software and
> totally eliminates having to invent a new way to "whiteout" to indicate
> removed variables (IIRC, it is valid to set a variable with no value).
>
> Further, once you find "the" env lump, the env processing is 100% the
> same so the implementation of my counterproposal simply wraps that
> existing env handling code with a "find the latest lump" loop.

i was saying what deltas would require.  i didnt say that is what i wanted to 
do.  i want to append the entire env blob.

> >> Since flash can change 1s to 0s, but cannot change them back, you could
> >> redefine the env storage to be
> >>    <next><crc><env><NUL>[undefined]
> >> (note the added <next> at the start).  By definition, the valid env's
> >> <next> == 0xFFFFFFFF.  To write a new env, simply write the offset of
> >> [undefined] into the <next> location and then write a new env lump.
> >>    <next><crc><env><NUL> <next><crc><env><NUL>[undefined]
> >>     ^^^^ offset of -----> ^^^^ 0xFFFFFFFF
> >
> > using a crc of 0x00000000 accomplishes the same thing and retains env
> > format
>
> Three problems:
> 1) 0x00000000 can be a valid CRC.  You either have a risk of failure
> (granted, very small) or you have to special case that instance.  Note
> that, if you don't handle the 0x00000000 case properly, you will
> probably brick the board.

i know 0x00000000 is a valid CRC.  the implied statement is that the next two 
bytes can tell you with clarity whether this is the environment or skipping 
it.  either way, i see it as a micro optimization that really isnt necessary 
for the larger idea.

> 3) My <next> offset is much faster since you don't have to step through
> the old env to find where the next one starts.  It also handles
> Wolfgang's point with NAND flashes needing to be aligned to blocks.
> (What do you do with a non-zero bad CRC?  Fall back to the redundant env?)

how much faster <next> is depends on the flash in use.  i dont think NAND can 
deal with this idea in general anyways since it has ECC checks in the OOB 
page.  unless the OOB can be erased independent of the data page ?  i thought 
erasing of NAND/OOB pages had to be done together.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20090413/63462203/attachment-0001.pgp 

[U-Boot] incremental environment updating

[Jerry Van Baren]

Wolfgang Denk wrote:
> Dear Jerry Van Baren,
> 
> In message <49E335BB.4050609@ge.com> you wrote:
>>> - it works only on NOR flash, not on any media that can be accessed
>>>   only in blocks
>> Not true: the offset merely needs to be aligned to the next block in 
>> that case:
>>    <next><crc><env><NUL>FFFFFFFFFFFF <next><crc><env><NUL>[undefined]
>>    ^^^^ offset of ------------------> ^^^^ 0xFFFFFFFF
>>
>> All flash that I can recall seeing allows at least a limited number of 
>> rewrites to a block (the limited number getting smaller on successive 
>> generations), so two writes to the env start block (once to write the 
>> new env, a second time to rewrite the 0xFFFFFFFF to point to the 
>> superceeding env).  If the flash didn't support two writes, that would 
>> be a show-stopper.  Due to how people use flash, I don't think the 
>> number will ever go to one write (IIRC, 4 is typical of current 
>> generation NAND).
> 
> So what do we save, then? Instead of a single write we now need
> actually two...

Well, I have a (NOR) flash with a 128Kbyte sector size.  If my env is 
~2K, I can do ~64 updates before I would need to erase the sector and 
start over.

 From a practical point of view, that probably isn't a big deal.  From a 
"bugs me" POV, it bugs me that I'm "wasting" 126K of that sector.  :-P

IIRC, redundant env lumps take two sectors so the redundant env isn't 
erased (need to check the code).  Depending on your acceptable risk 
tolerance, the "redundant" env lump would simply be lump(n-1).

[snip]

Best regards,
gvb