Bitbake task logging

[Richard Purdie <richard.purdie@linuxfoundation.org>]

I've been looking at the changes in bitbake master and those in Poky.
Both are trying to improve the logging situation but I think we need to
discuss/agree what we're trying to achieve.

I'm trying to think about this from the user point of view. What does a
user need and expect from bitbake to be as easy to use as possible and
figure out whats going on. New users often ask "what is it doing? what
functions does it run?" and its hard to work out for someone new to it.

Some of the list below works today, some is close, some doesn't but I'm
looking to work out what the experience should be, then we can figure
out how to get there technically.

The things I think we need are:

a) A logfile per task.

b) All output from the task should end up in the logfile, copied or
otherwise, be it bitbake LogRecords, output to stdout, stderr or
anything else.

c) The logfile should be able to include different output from that
shown in the UI console. For example, I think "note" output in the
logfiles might be useful all the time, I'm wondering if it ever is
useful on the UI console. Turning off "note" for the console but keeping
it and maybe using it more in tasks could be useful. For example,
exec_func() could note which functions are being executed and I can
imagine a user finding that very useful in the log file for debugging
(which is when a user would look there). Debug output is probably too
verbose even for the log files in general but I'm open to persuasion one
way or another on that.

d) Handling of the default stdout is a tricky one. I think it makes most
sense to redirect this to the logfile always, for both python and shell 
tasks. For bitbake itself, nothing in bitbake should be putting output
there though IMO. This makes sense since we can control stdout from
within bitbake itself but from within generic tasks we cannot make
guarantees. Python tasks may make os.system calls, or run binaries and
it would make sense for stdout to just work for these rather than need
special handling or wrapper functions especially for log handling (they
could be a good idea for other reasons).

e) I'm not 100% on this yet but I'm wondering if we should scrap
stdout/stderr output to the UI console in debug mode, just make it clear
where the output went on the console? If you ever have two tasks running
the output is unreadable anyway.

Thoughts/Comments/Suggestions?

Cheers,

Richard