What's this

What's this

[Gary Thomas]

I just updated my Poky repo and did a rebuild and saw this "error"
(although it carried on with the build):
ERROR: base-files-3.0.14-r89 do_install: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 
44f498452dfce5875451bca66b230e31 for /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
ERROR: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 44f498452dfce5875451bca66b230e31 for 
/local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install

What does this mean?  Should I be worried?

meta              = "master:04319c0340010c4e0f78cab252cfcb0ebe23ee84"

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: What's this

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

On 5 June 2016 at 17:44, Gary Thomas <gary@mlbassoc.com> wrote:

> I just updated my Poky repo and did a rebuild and saw this "error"
> (although it carried on with the build):
> ERROR: base-files-3.0.14-r89 do_install: Taskhash mismatch
> ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 44f498452dfce5875451bca66b230e31
> for
> /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
> ERROR: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses
> 44f498452dfce5875451bca66b230e31 for
> /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
>
> What does this mean?  Should I be worried?
>

It means the hash calculated my the bitbake master was different to the
hash calculated when the worker started up.  This usually means that you're
using something like ${TIME} in the recipe but not marking it appropriatly
so the cache ignores it.  Do you have a base-files bbappend that writes a
timestamp?

Ross

[-- Attachment #2: Type: text/html, Size: 1396 bytes --]

Re: What's this

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com> wrote:

> It means the hash calculated my the bitbake master was different to the
> hash calculated when the worker started up.  This usually means that you're
> using something like ${TIME} in the recipe but not marking it appropriatly
> so the cache ignores it.  Do you have a base-files bbappend that writes a
> timestamp?
>

The always wise Joshua reminds me that if your DISTRO_VERSION contains
${DATETIME} then this happens.  If you're doing this then you'll want to
set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from getting
into the cache (or not put the current date/time into the distro version).

Ross

[-- Attachment #2: Type: text/html, Size: 1064 bytes --]

Re: What's this

[Gary Thomas]

On 2016-06-07 18:20, Burton, Ross wrote:
>
> On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com <mailto:ross.burton@intel.com>> wrote:
>
>     It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up.
>     This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the
>     cache ignores it.  Do you have a base-files bbappend that writes a timestamp?
>
>
> The always wise Joshua reminds me that if your DISTRO_VERSION contains ${DATETIME} then this happens.  If you're doing
> this then you'll want to set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from getting into the cache (or not
> put the current date/time into the distro version).
>

Ah, that's probably it.  Exactly how do I set that exclusion?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: What's this

[Joshua G Lock]

On Tue, 2016-06-07 at 18:58 +0200, Gary Thomas wrote:
> On 2016-06-07 18:20, Burton, Ross wrote:
> > 
> > On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com
> > <mailto:ross.burton@intel.com>> wrote:
> > 
> >     It means the hash calculated my the bitbake master was
> > different to the hash calculated when the worker started up.
> >     This usually means that you're using something like ${TIME} in
> > the recipe but not marking it appropriatly so the
> >     cache ignores it.  Do you have a base-files bbappend that
> > writes a timestamp?
> > 
> > 
> > The always wise Joshua reminds me that if your DISTRO_VERSION
> > contains ${DATETIME} then this happens.  If you're doing
> > this then you'll want to set [vardepsexclude] on DISTRO_VERSION to
> > stop the DATETIME from getting into the cache (or not
> > put the current date/time into the distro version).
> > 
> 
> Ah, that's probably it.  Exactly how do I set that exclusion?

Assuming DISTRO_VERSION includes DATETIME

    DISTRO_VERSION[vardepsexclude] = "DATETIME"

see for example:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta-poky/conf/dist
ro/poky.conf#n14

Regards,

Joshua

Re: What's this

[Paul Eggleton]

On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
> On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com> wrote:
> > It means the hash calculated my the bitbake master was different to the
> > hash calculated when the worker started up.  This usually means that
> > you're
> > using something like ${TIME} in the recipe but not marking it appropriatly
> > so the cache ignores it.  Do you have a base-files bbappend that writes a
> > timestamp?
> 
> The always wise Joshua reminds me that if your DISTRO_VERSION contains
> ${DATETIME} then this happens.  If you're doing this then you'll want to
> set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from getting
> into the cache (or not put the current date/time into the distro version).

I think we need to handle this situation better - if it's really worth 
producing an error about then it's worth producing an error message that 
people can actually understand, particularly as it's recently added 
validation.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: What's this

[Richard Purdie]

On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
> On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
> > On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
> > wrote:
> > > It means the hash calculated my the bitbake master was different
> > > to the
> > > hash calculated when the worker started up.  This usually means
> > > that
> > > you're
> > > using something like ${TIME} in the recipe but not marking it
> > > appropriatly
> > > so the cache ignores it.  Do you have a base-files bbappend that
> > > writes a
> > > timestamp?
> > 
> > The always wise Joshua reminds me that if your DISTRO_VERSION
> > contains
> > ${DATETIME} then this happens.  If you're doing this then you'll
> > want to
> > set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from
> > getting
> > into the cache (or not put the current date/time into the distro
> > version).
> 
> I think we need to handle this situation better - if it's really
> worth 
> producing an error about then it's worth producing an error message
> that 
> people can actually understand, particularly as it's recently added 
> validation.

It was silently running into problems due to this all along but not
reporting it. Its now reporting it which is better than silently things
behaving strangely.

Its very hard for bitbake to know why the hashes differ, it only knows
the values afterwards and hence that they've changed, the information
about how that hash was constructed is not present in any of bitbake's
caches. That implies to have better messages we need to write out more
data.

I did add a patch to make bitbake write out data to allow
reconstruction of basehash (which is part of taskhash). Sadly the
parsing performance was diabolical (10 times slower). I think that
could perhaps be improved if the files don't require an atomic move
during creation but I haven't had time to look further at it.

So whilst I do agree, what is the price people are willing to pay to
have those better messages?

Cheers,

Richard

Re: What's this

[Paul Eggleton]

On Tue, 07 Jun 2016 23:20:26 Richard Purdie wrote:
> On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
> > On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
> > > On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
> > > wrote:
> > > > It means the hash calculated my the bitbake master was different
> > > > to the hash calculated when the worker started up.  This usually means
> > > > that you're using something like ${TIME} in the recipe but not marking 
> > > > it appropriatly so the cache ignores it.  Do you have a base-files
> > > > bbappend that writes a timestamp?
> > > 
> > > The always wise Joshua reminds me that if your DISTRO_VERSION
> > > contains ${DATETIME} then this happens.  If you're doing this then
> > > you'll want to set [vardepsexclude] on DISTRO_VERSION to stop the
> > > DATETIME from getting into the cache (or not put the current date/time
> > > into the distro version).
> > 
> > I think we need to handle this situation better - if it's really
> > worth producing an error about then it's worth producing an error message
> > that people can actually understand, particularly as it's recently added
> > validation.
> 
> It was silently running into problems due to this all along but not
> reporting it. Its now reporting it which is better than silently things
> behaving strangely.

What are the actual consequences in a situation like this where we have 
something like ${DATETIME} in another variable referenced by the task?

I haven't looked at all of the code involved (and that's probably the root of 
my problem) but I don't quite understand how this is coming about. DATE and 
TIME are supposed to be determined at the start of the build and explicitly 
passed to the worker, so they don't change during the build. What am I 
missing?
 
> Its very hard for bitbake to know why the hashes differ, it only knows
> the values afterwards and hence that they've changed, the information
> about how that hash was constructed is not present in any of bitbake's
> caches. That implies to have better messages we need to write out more
> data.
> 
> I did add a patch to make bitbake write out data to allow
> reconstruction of basehash (which is part of taskhash). Sadly the
> parsing performance was diabolical (10 times slower). I think that
> could perhaps be improved if the files don't require an atomic move
> during creation but I haven't had time to look further at it.
> 
> So whilst I do agree, what is the price people are willing to pay to
> have those better messages?

Clearly a 10x slowdown is unacceptable, hopefully we can find another way of 
dealing with this. I guess if we're able to do nothing else a brief 
explanation of what to look for (i.e. variables that might change with time) 
in the error message would be helpful, but I hope we can do better than that.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: What's this

[Richard Purdie]

On Wed, 2016-06-08 at 20:59 +1200, Paul Eggleton wrote:
> On Tue, 07 Jun 2016 23:20:26 Richard Purdie wrote:
> > On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
> > > On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
> > > > On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
> > > > wrote:
> > > > > It means the hash calculated my the bitbake master was
> > > > > different
> > > > > to the hash calculated when the worker started up.  This
> > > > > usually means
> > > > > that you're using something like ${TIME} in the recipe but
> > > > > not marking 
> > > > > it appropriatly so the cache ignores it.  Do you have a base
> > > > > -files
> > > > > bbappend that writes a timestamp?
> > > > 
> > > > The always wise Joshua reminds me that if your DISTRO_VERSION
> > > > contains ${DATETIME} then this happens.  If you're doing this
> > > > then
> > > > you'll want to set [vardepsexclude] on DISTRO_VERSION to stop
> > > > the
> > > > DATETIME from getting into the cache (or not put the current
> > > > date/time
> > > > into the distro version).
> > > 
> > > I think we need to handle this situation better - if it's really
> > > worth producing an error about then it's worth producing an error
> > > message
> > > that people can actually understand, particularly as it's
> > > recently added
> > > validation.
> > 
> > It was silently running into problems due to this all along but not
> > reporting it. Its now reporting it which is better than silently
> > things
> > behaving strangely.
> 
> What are the actual consequences in a situation like this where we 
> have something like ${DATETIME} in another variable referenced by the
> task?

A mismatch means the cached internal values bitbake is using on the
cooker/server side don't match what the worker is actually doing. Its
hard to say for sure exactly what impact that would have.

If for example its in a do_build task, it means that some invalid data
may be written out to the do_build sigdata file and confuse later build
analysis based on those files. You can usually tell such a file by the
fact its named with one hash name but when you compute the value from
the entries in the file, you get a different hash.

If its a more important task like do_install, it could mean something
doesn't rerun when it should, or vice versa and the effect of a corrupt
sidinfo/sigdata is more significant.

With the changes to error in these cases, we get to know about cases of
mismatch and we went and fixed the majority of causes of these. I
believe those mismatches were in part at fault for things like diffsigs
not working as expected in some cases.

> I haven't looked at all of the code involved (and that's probably the
> root of my problem) but I don't quite understand how this is coming
> about. DATE and TIME are supposed to be determined at the start of
> the build and explicitly passed to the worker, so they don't change
> during the build. What am I missing?

Bitbake computes the hashes at parse time and caches them. At run time,
it then builds a new data store in the worker. These errors come about
when the values in the cache from parsing don't match the runtime
execution environment.
 
> > Its very hard for bitbake to know why the hashes differ, it only
> > knows
> > the values afterwards and hence that they've changed, the
> > information
> > about how that hash was constructed is not present in any of
> > bitbake's
> > caches. That implies to have better messages we need to write out
> > more
> > data.
> > 
> > I did add a patch to make bitbake write out data to allow
> > reconstruction of basehash (which is part of taskhash). Sadly the
> > parsing performance was diabolical (10 times slower). I think that
> > could perhaps be improved if the files don't require an atomic move
> > during creation but I haven't had time to look further at it.
> > 
> > So whilst I do agree, what is the price people are willing to pay
> > to
> > have those better messages?
> 
> Clearly a 10x slowdown is unacceptable, hopefully we can find another
> way of 
> dealing with this. I guess if we're able to do nothing else a brief 
> explanation of what to look for (i.e. variables that might change
> with time) 
> in the error message would be helpful, but I hope we can do better
> than that.

So do I, its a time problem on my part to look into it. As mentioned
above, I believe this problem was meaning diffsigs was unreliable and
it was having other unknown impact on builds rebuilding when they
shouldn't or vice versa. I believe those are important to fix. We need
better sstate hash debugging/telemetry but exactly how to do this
without performance impact remains to be seen. I'm continue to think on
it (along with too many other issues) but am open to ideas.

Cheers,

Richard

Re: What's this

[Mike Looijmans]

On 08-06-16 00:20, Richard Purdie wrote:
> On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
>> On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
>>> On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
>>> wrote:
>>>> It means the hash calculated my the bitbake master was different
>>>> to the
>>>> hash calculated when the worker started up.  This usually means
>>>> that
>>>> you're
>>>> using something like ${TIME} in the recipe but not marking it
>>>> appropriatly
>>>> so the cache ignores it.  Do you have a base-files bbappend that
>>>> writes a
>>>> timestamp?
>>>
>>> The always wise Joshua reminds me that if your DISTRO_VERSION
>>> contains
>>> ${DATETIME} then this happens.  If you're doing this then you'll
>>> want to
>>> set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from
>>> getting
>>> into the cache (or not put the current date/time into the distro
>>> version).
>>
>> I think we need to handle this situation better - if it's really
>> worth
>> producing an error about then it's worth producing an error message
>> that
>> people can actually understand, particularly as it's recently added
>> validation.
>
> It was silently running into problems due to this all along but not
> reporting it. Its now reporting it which is better than silently things
> behaving strangely.
>
> Its very hard for bitbake to know why the hashes differ, it only knows
> the values afterwards and hence that they've changed, the information
> about how that hash was constructed is not present in any of bitbake's
> caches. That implies to have better messages we need to write out more
> data.
>
> I did add a patch to make bitbake write out data to allow
> reconstruction of basehash (which is part of taskhash). Sadly the
> parsing performance was diabolical (10 times slower). I think that
> could perhaps be improved if the files don't require an atomic move
> during creation but I haven't had time to look further at it.
>
> So whilst I do agree, what is the price people are willing to pay to
> have those better messages?

I have a build machine that every so often runs into "basehash mismatch" 
problems (but never when you want it to, such as now), so getting some 
information on what might cause it would be "priceless". I'd happily let the 
machine run for 10 days straight if at the end I get a message that I can work 
with.

Which implies that any performance hit is acceptable, if there's a switch to 
enable and disable that extra diagnostic. Oh dear, just what we need, yet 
another switch...

M.



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: What's this

[Gary Thomas]

On 2016-06-08 11:08, Mike Looijmans wrote:
> On 08-06-16 00:20, Richard Purdie wrote:
>> On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
>>> On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
>>>> On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
>>>> wrote:
>>>>> It means the hash calculated my the bitbake master was different
>>>>> to the
>>>>> hash calculated when the worker started up.  This usually means
>>>>> that
>>>>> you're
>>>>> using something like ${TIME} in the recipe but not marking it
>>>>> appropriatly
>>>>> so the cache ignores it.  Do you have a base-files bbappend that
>>>>> writes a
>>>>> timestamp?
>>>>
>>>> The always wise Joshua reminds me that if your DISTRO_VERSION
>>>> contains
>>>> ${DATETIME} then this happens.  If you're doing this then you'll
>>>> want to
>>>> set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from
>>>> getting
>>>> into the cache (or not put the current date/time into the distro
>>>> version).
>>>
>>> I think we need to handle this situation better - if it's really
>>> worth
>>> producing an error about then it's worth producing an error message
>>> that
>>> people can actually understand, particularly as it's recently added
>>> validation.
>>
>> It was silently running into problems due to this all along but not
>> reporting it. Its now reporting it which is better than silently things
>> behaving strangely.
>>
>> Its very hard for bitbake to know why the hashes differ, it only knows
>> the values afterwards and hence that they've changed, the information
>> about how that hash was constructed is not present in any of bitbake's
>> caches. That implies to have better messages we need to write out more
>> data.
>>
>> I did add a patch to make bitbake write out data to allow
>> reconstruction of basehash (which is part of taskhash). Sadly the
>> parsing performance was diabolical (10 times slower). I think that
>> could perhaps be improved if the files don't require an atomic move
>> during creation but I haven't had time to look further at it.
>>
>> So whilst I do agree, what is the price people are willing to pay to
>> have those better messages?
>
> I have a build machine that every so often runs into "basehash mismatch" problems (but never when you want it to, such
> as now), so getting some information on what might cause it would be "priceless". I'd happily let the machine run for 10
> days straight if at the end I get a message that I can work with.
>
> Which implies that any performance hit is acceptable, if there's a switch to enable and disable that extra diagnostic.
> Oh dear, just what we need, yet another switch...
>

I wonder if it could be something as simple as an NTP time update
which happens periodically on my build machine?  Since I've only
ever seen this message _once_ after literally thousands of builds,
it is a rare bird indeed!

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: What's this

[Julien Gueytat]

I got that taskhash mismatch too and much more often than once per 
million. ;) I'm quite happy that you're discussing it here as I did not 
know what it was.

Re-running the task a second time in a row makes the warning disappearing.

Regards,

Le 08/06/2016 11:18, Gary Thomas a écrit :
> On 2016-06-08 11:08, Mike Looijmans wrote:
>> On 08-06-16 00:20, Richard Purdie wrote:
>>> On Wed, 2016-06-08 at 09:24 +1200, Paul Eggleton wrote:
>>>> On Tue, 07 Jun 2016 17:20:12 Burton, Ross wrote:
>>>>> On 7 June 2016 at 17:02, Burton, Ross <ross.burton@intel.com>
>>>>> wrote:
>>>>>> It means the hash calculated my the bitbake master was different
>>>>>> to the
>>>>>> hash calculated when the worker started up.  This usually means
>>>>>> that
>>>>>> you're
>>>>>> using something like ${TIME} in the recipe but not marking it
>>>>>> appropriatly
>>>>>> so the cache ignores it.  Do you have a base-files bbappend that
>>>>>> writes a
>>>>>> timestamp?
>>>>>
>>>>> The always wise Joshua reminds me that if your DISTRO_VERSION
>>>>> contains
>>>>> ${DATETIME} then this happens.  If you're doing this then you'll
>>>>> want to
>>>>> set [vardepsexclude] on DISTRO_VERSION to stop the DATETIME from
>>>>> getting
>>>>> into the cache (or not put the current date/time into the distro
>>>>> version).
>>>>
>>>> I think we need to handle this situation better - if it's really
>>>> worth
>>>> producing an error about then it's worth producing an error message
>>>> that
>>>> people can actually understand, particularly as it's recently added
>>>> validation.
>>>
>>> It was silently running into problems due to this all along but not
>>> reporting it. Its now reporting it which is better than silently things
>>> behaving strangely.
>>>
>>> Its very hard for bitbake to know why the hashes differ, it only knows
>>> the values afterwards and hence that they've changed, the information
>>> about how that hash was constructed is not present in any of bitbake's
>>> caches. That implies to have better messages we need to write out more
>>> data.
>>>
>>> I did add a patch to make bitbake write out data to allow
>>> reconstruction of basehash (which is part of taskhash). Sadly the
>>> parsing performance was diabolical (10 times slower). I think that
>>> could perhaps be improved if the files don't require an atomic move
>>> during creation but I haven't had time to look further at it.
>>>
>>> So whilst I do agree, what is the price people are willing to pay to
>>> have those better messages?
>>
>> I have a build machine that every so often runs into "basehash 
>> mismatch" problems (but never when you want it to, such
>> as now), so getting some information on what might cause it would be 
>> "priceless". I'd happily let the machine run for 10
>> days straight if at the end I get a message that I can work with.
>>
>> Which implies that any performance hit is acceptable, if there's a 
>> switch to enable and disable that extra diagnostic.
>> Oh dear, just what we need, yet another switch...
>>
>
> I wonder if it could be something as simple as an NTP time update
> which happens periodically on my build machine?  Since I've only
> ever seen this message _once_ after literally thousands of builds,
> it is a rare bird indeed!
>

Re: What's this

[Gary Thomas]

On 2016-06-07 18:02, Burton, Ross wrote:
>
> On 5 June 2016 at 17:44, Gary Thomas <gary@mlbassoc.com <mailto:gary@mlbassoc.com>> wrote:
>
>     I just updated my Poky repo and did a rebuild and saw this "error"
>     (although it carried on with the build):
>     ERROR: base-files-3.0.14-r89 do_install: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses
>     44f498452dfce5875451bca66b230e31 for
>     /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
>     ERROR: Taskhash mismatch ceb25febc6f11a5cc2b3c4a4b0b2c136 verses 44f498452dfce5875451bca66b230e31 for
>     /local/poky-cutting-edge/meta/recipes-core/base-files/base-files_3.0.14.bb.do_install
>
>     What does this mean?  Should I be worried?
>
>
> It means the hash calculated my the bitbake master was different to the hash calculated when the worker started up.
> This usually means that you're using something like ${TIME} in the recipe but not marking it appropriatly so the cache
> ignores it.  Do you have a base-files bbappend that writes a timestamp?

Sorry, nothing like that in my layers.  Note: this only happened once,
immediately after a poky pull and the builds have been happy since.
Whatever it was only happened that one time (not that that should be
any less concerning...)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------