[U-Boot] [RFD] store env size in env area (+ cookie + len)

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Frans Meulenbroeks]

I've seen a few times that people get a u-boot with their board and
want to use fw_setenv under linux to tweak with environment variables.
This requires them to come up with an fw-env.config (if not provided).
One of the things that I've seen, that some people have trouble with
it is obtaining the proper size of the environment area to be stuffed
in that file.

This is something that could be resolved quite easily by adding the
size of the environment area to the header preceding the environment
(which now contains a checksum and, in case of redundant areas, a flag
byte)

Pro: no more need to give sizes in fw-env.config
Con: uses a 2-4 more bytes (personally I'd say size is 4 bytes just to be safe)
Con: not compatible with current env layout so (without precautions) a
new u-boot cannot  use the old env and vice versa.

If compatibility is deemed important, we could extend the header also
with a magic cookie and a version number. If absent it is an old style
env, if present it is a new style env. Version number then can also be
used should there be future changes in the env layout.
(if compatibility is not deemed important of course this is not needed).

Btw thinking about cookies: if the env were to have a cookie in either
the header or e.g. as first env var, fw_setenv would not even need
fw-env.config at all. It could just seek in /dev/mtd (I'm assuming env
in flash here) for the cookie.
This seeking should not take too much time if we can assume (or have a
config var) a certain alignment (e.g. on flash page size). After
finding the cookie of course still a crc check would be done to
validate the area.

If desired I can do some implementation work for this and submit a
patch, but before investing time in it I would like to find out if
there is actual interest in this and if this is considered useful.

Summarizing the questions:
- is it desirable to have the env length added to the env header
- is it desirable to have a cookie added to the env header
- is it desirable to have a version number added to the env header

Best regards, Frans.

PS: if we are going to touch the env header, I would suggest to always
have the redundant flag in the header even if there is only  one
environment. It will probably make the code to load the env a little
bit simpler).

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Reinhard Meyer]

Dear Frans Meulenbroeks,
> Pro: no more need to give sizes in fw-env.config
> Con: uses a 2-4 more bytes (personally I'd say size is 4 bytes just to be safe)
That's less than any variable takes in the env, so no problem.
> Con: not compatible with current env layout so (without precautions) a
> new u-boot cannot  use the old env and vice versa.
When we put a new u-Boot on a hardware, we erase the environment anyway.
But that might affect other systems that rely on storing vital product data
like MAC addresses or Serialnumbers in the environment.
> If compatibility is deemed important, we could extend the header also
> with a magic cookie and a version number. If absent it is an old style
> env, if present it is a new style env. Version number then can also be
> used should there be future changes in the env layout.
> (if compatibility is not deemed important of course this is not needed).
See above, might be needed. Should be an option. With the new hashed
environment it should not be complicated to do.
> Summarizing the questions:
> - is it desirable to have the env length added to the env header
Yes
> - is it desirable to have a cookie added to the env header
Yes
> - is it desirable to have a version number added to the env header
Yes. If the Version number mismatches, the default env should
be used (config option)
> PS: if we are going to touch the env header, I would suggest to always
> have the redundant flag in the header even if there is only  one
> environment. It will probably make the code to load the env a little
> bit simpler).
Yes. The env header should NOT have any #ifdefs anymore. Check the
new environment handling by Wolfgang, your work should be based on that,
I suggest.

Reinhard

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Matthias Weißer]

Am 12.08.2010 08:43, schrieb Frans Meulenbroeks:
 > I've seen a few times that people get a u-boot with their board and
 > want to use fw_setenv under linux to tweak with environment variables.
 > This requires them to come up with an fw-env.config (if not provided).
 > One of the things that I've seen, that some people have trouble with
 > it is obtaining the proper size of the environment area to be stuffed
 > in that file.

 > Con: not compatible with current env layout so (without precautions) a
 > new u-boot cannot  use the old env and vice versa.

I didn't dig into the code but I don't think it is good idea to change 
the environment format. There may be (or at least there is for my board) 
code that scans the environment which is not derived from u-boot code. A 
new header entry would break this code.

Why not store the size of the environment sector as env variable? The 
size should be only needed for storing the environment and not for 
loading (as it is terminated by a '\0'). So we could load the 
environment, check for "env_size" and if its not there take the value 
from fw_env.config

Matthias

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Alexander Stein]

On Thursday 12 August 2010, 09:27:31 you wrote:
> Why not store the size of the environment sector as env variable? The
> size should be only needed for storing the environment and not for
> loading (as it is terminated by a '\0'). So we could load the
> environment, check for "env_size" and if its not there take the value
> from fw_env.config

Another possibility is to use an own mtd partition for env storage. This way 
you shouldn't need to know any size or offset at all.

Best regards,
Alexander

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear Alexander Stein,

In message <201008121007.20433.alexander.stein@systec-electronic.com> you wrote:
> On Thursday 12 August 2010, 09:27:31 you wrote:
> > Why not store the size of the environment sector as env variable? The
> > size should be only needed for storing the environment and not for
> > loading (as it is terminated by a '\0'). So we could load the
> > environment, check for "env_size" and if its not there take the value
> > from fw_env.config
> 
> Another possibility is to use an own mtd partition for env storage. This way 
> you shouldn't need to know any size or offset at all.

The size of the environment is typically smaller than the sector size
and thus than the MTD partition size.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Nothing ever becomes real till it is experienced -- even a proverb is
no proverb to you till your life has illustrated it.     - John Keats

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Matthias Weißer]

Am 12.08.2010 09:54, schrieb Frans Meulenbroeks:
> 2010/8/12 Matthias Wei?er<weisserm@arcor.de>:
>> Why not store the size of the environment sector as env variable? The size
>> should be only needed for storing the environment and not for loading (as it
>> is terminated by a '\0'). So we could load the environment, check for
>> "env_size" and if its not there take the value from fw_env.config
>
> Currently the crc of the environment is calculated and if it does not
> match with the stored crc the default environment is used.
> For determining the crc the size is needed so storing the size in the
> env creates some form of catch-22. Of course the code could be
> modified to first peek at the unverified env to find the size.
> That is a different approach. Imho not too neat.

I see. I was thinking that the CRC is only calculated over the 
environment until the '\0' end mark is reached. But after checking my 
environment code I see that the CRC is calculated over the whole sector.

> Wrt the format: I understood from the mail from Reinhard that the way
> the env is stored is going to change anyway. I saw the patches of
> Wolfgang pass by, but didn't have time to study them yet.

I also didn't go that deep into Wolfgangs patches but from what I 
understand the "on disk format" will not change. Will it?

Maybe the size could be added at the end of the environment (behind the 
'\0'). So we wouldn't loose compatibility.

Matthias

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Reinhard Meyer]

Dear Matthias Wei?er,
>> Wrt the format: I understood from the mail from Reinhard that the way
>> the env is stored is going to change anyway. I saw the patches of
>> Wolfgang pass by, but didn't have time to study them yet.
What I meant, is, that with those patches and the environment in RAM
not being a 1:1 copy of what is stored "on disk" anymore, its easy to
write it in any format you desire onto "disk".

> I also didn't go that deep into Wolfgangs patches but from what I 
> understand the "on disk format" will not change. Will it?
No its not changed. Only the format in RAM is changed to a hashed list.
Therefore it would not matter how it is stored on external media since
a "read" and "write" function is needed anyway.

You only need to change/rewrite those functions to store the environment
in any fashion you like...

Reinhard

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Reinhard Meyer]

Thinking more about it:

> You only need to change/rewrite those functions to store the environment
> in any fashion you like...

The header could also include the current length of the env,
not just the max.

And env in EEPROM would benefit in speed if only the used space
is written/read. A 16k ENV area in I2C EEPROM can take a few
seconds to save if not tuned to max. speed.
If the ENV uses just a few 100 Bytes, which it
normally does, it would speed up saving (and reading).

<duck> one could also save the env in XML format... </duck>

Reinhard

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear Reinhard Meyer,

In message <4C63B262.8070004@emk-elektronik.de> you wrote:
> 
> And env in EEPROM would benefit in speed if only the used space
> is written/read. A 16k ENV area in I2C EEPROM can take a few
> seconds to save if not tuned to max. speed.
> If the ENV uses just a few 100 Bytes, which it
> normally does, it would speed up saving (and reading).

This is an improvement that can be implemented independent from other
changes discussed here. Patches are welcome.

> <duck> one could also save the env in XML format... </duck>

Yes, or as a SQLite data base. There are many options. Not all of them
are really clever ones, though.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Inside every old person is a young person wondering what happened.
                              - Terry Pratchett, _Moving Pictures_

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear Reinhard Meyer,

In message <4C63AE37.1020206@emk-elektronik.de> you wrote:
>
> > understand the "on disk format" will not change. Will it?
> No its not changed. Only the format in RAM is changed to a hashed list.
> Therefore it would not matter how it is stored on external media since
> a "read" and "write" function is needed anyway.
> 
> You only need to change/rewrite those functions to store the environment
> in any fashion you like...

Please spell "compatibility".

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
In accord with UNIX philosophy, Perl gives you enough  rope  to  hang
yourself.              - L. Wall & R. L. Schwartz, _Programming Perl_

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Reinhard Meyer]

Dear Wolfgang Denk,
>> You only need to change/rewrite those functions to store the environment
>> in any fashion you like...
> 
> Please spell "compatibility".

Hmm, I think all IBM(R) PC(R) USERS(R) have spelled that alot.
Where this lead to, everyone knows...
Though its nice that some Win95 Programs still run under Vista ;)

And who says the "different" storage could not be an
configuration option?

Reinhard

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear Reinhard Meyer,

In message <4C63B8E6.8030805@emk-elektronik.de> you wrote:
>
> Hmm, I think all IBM(R) PC(R) USERS(R) have spelled that alot.
> Where this lead to, everyone knows...
> Though its nice that some Win95 Programs still run under Vista ;)
> 
> And who says the "different" storage could not be an
> configuration option?

I really fail to see any real advantages either. The existing format
has the benefit of being simple and efficient for the purposes it has
been designed for.

If you want different output formats, you can add more data formats to
the "export" and "import" options of the new environment handling
code, but you also would have to add bootstrap code to read this format
from early environment (i. e. without data  or bss segments, and with
minimal stack size).

My feeling is that the result would be complicated and inefficient.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
What is mind?  No matter.  What is matter?  Never mind.
                                      -- Thomas Hewitt Key, 1799-1875

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear =?ISO-8859-1?Q?Matthias_Wei=DFer?=,

In message <4C63ACA6.6080303@arcor.de> you wrote:
> Am 12.08.2010 09:54, schrieb Frans Meulenbroeks:
> > 2010/8/12 Matthias Wei?er<weisserm@arcor.de>:
> >> Why not store the size of the environment sector as env variable? The size
> >> should be only needed for storing the environment and not for loading (as it
> >> is terminated by a '\0'). So we could load the environment, check for
> >> "env_size" and if its not there take the value from fw_env.config
> >
> > Currently the crc of the environment is calculated and if it does not
> > match with the stored crc the default environment is used.
> > For determining the crc the size is needed so storing the size in the
> > env creates some form of catch-22. Of course the code could be
> > modified to first peek at the unverified env to find the size.
> > That is a different approach. Imho not too neat.

Was this an off-list reply? I cannot see any trace of this message on
the list.

Please always keep the list on Cc: !

> I see. I was thinking that the CRC is only calculated over the
> environment until the '\0' end mark is reached. But after checking my
> environment code I see that the CRC is calculated over the whole sector.

For the sake of simplicity it should be possible to assume that the
CRC is correct, look up the env_size variable, and then check the CRC
with the length thus found.

> > Wrt the format: I understood from the mail from Reinhard that the way
> > the env is stored is going to change anyway. I saw the patches of
> > Wolfgang pass by, but didn't have time to study them yet.

The external storage format will NOT change. Compatibility with
existing code (not only in U-Boot) is a major concern of all such
changes.

> I also didn't go that deep into Wolfgangs patches but from what I
> understand the "on disk format" will not change. Will it?

No, it ill remain exactly the same.

> Maybe the size could be added at the end of the environment (behind the
> '\0'). So we wouldn't loose compatibility.

It would add new issues, like when the envrionment is filing up.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Change is the essential process of all existence.
	-- Spock, "Let That Be Your Last Battlefield",
	   stardate 5730.2

[U-Boot] [RFD] store env size in env area (+ cookie + len)

[Wolfgang Denk]

Dear Frans Meulenbroeks,

In message <AANLkTi=+67z34H+R+tNab9MOPa55oLoRW0Vvs9BEEbf+@mail.gmail.com> you wrote:
> I've seen a few times that people get a u-boot with their board and
> want to use fw_setenv under linux to tweak with environment variables.
> This requires them to come up with an fw-env.config (if not provided).
> One of the things that I've seen, that some people have trouble with
> it is obtaining the proper size of the environment area to be stuffed
> in that file.

This should be no problem, as a short look at the board config header
file should be enough to provide all details you need.

> This is something that could be resolved quite easily by adding the
> size of the environment area to the header preceding the environment
> (which now contains a checksum and, in case of redundant areas, a flag
> byte)
> 
> Pro: no more need to give sizes in fw-env.config
> Con: uses a 2-4 more bytes (personally I'd say size is 4 bytes just to be safe)
> Con: not compatible with current env layout so (without precautions) a
> new u-boot cannot  use the old env and vice versa.

The incompatibility with the existing code IMHO is a killing point for
such a suggestion.

> If compatibility is deemed important, we could extend the header also
> with a magic cookie and a version number. If absent it is an old style
> env, if present it is a new style env. Version number then can also be
> used should there be future changes in the env layout.
> (if compatibility is not deemed important of course this is not needed).

I think you make this unessesarily complicated. Why not just use an
envrionment variable ("env_size") that gets automatically added,
similarly to what we do with "vers" ?

This allows to make it easily optional, and does not create any
incompatibility issues.

> Btw thinking about cookies: if the env were to have a cookie in either
> the header or e.g. as first env var, fw_setenv would not even need
> fw-env.config at all. It could just seek in /dev/mtd (I'm assuming env
> in flash here) for the cookie.
> This seeking should not take too much time if we can assume (or have a
> config var) a certain alignment (e.g. on flash page size). After
> finding the cookie of course still a crc check would be done to
> validate the area.

Heh. Assume the environment is stored in a 64 GB NAND flash device.
Guess how long the seeking might take, and how many false positives
you might find.

> Summarizing the questions:
> - is it desirable to have the env length added to the env header
> - is it desirable to have a cookie added to the env header
> - is it desirable to have a version number added to the env header

NAK to all of these, because of the incompatibility issues.

> PS: if we are going to touch the env header, I would suggest to always
> have the redundant flag in the header even if there is only  one
> environment. It will probably make the code to load the env a little
> bit simpler).

I have something in mind which eliminates the need for this flag
alltogether, including the need to always unlock and write to both
copies; it should also allow to add more copies so you could have more
than 2 generations of the environment.  I have an implementation in
mind, but I want to get the new environment code get into mainline
first.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The more we disagree, the more chance there is that at least  one  of
us is right.