Re: [PATCH] classes/rm_work: use the idle I/O scheduler class

[Patrick Ohly <patrick.ohly@intel.com>]

On Tue, 2016-06-14 at 16:18 +0100, Ross Burton wrote:
> As rm_work is just cleanup it shouldn't starve more important tasks such as
> do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
> scheduler class.

Whether that's desirable depends a lot on the goals for rm_work: when I
tried to use it for TravisCI to get around some pretty tight disk space
constraints, I found that do_rm_work was often not scheduled early
enough because other tasks generating more files had higher priority.

Reducing the IO priority of do_rm_work may have the same effect: it
runs, but then instead of removing files, the system produces more of
them, thus increasing the risk of exhausting the disk space.

I suspect a lot of benchmarking will be needed to determine what really
works well and what doesn't.

I ended up writing a custom scheduler for running under TravisCI:
https://github.com/01org/meta-intel-iot-security/blob/master/scripts/rmwork.py

That orders do_rm_work before any other task and also orders all tasks
related to single recipe so that they run together, thus making it
possible to clean up after do_build sooner. As an additional tweak it
distinguishes between "compile" and "cleanup" tasks and can run
"cleanup" tasks when the normal scheduler wouldn't because
BB_NUMBER_THREADS is reached.

But it has the same problem: not enough benchmarking to really quantify
the effect. All I know is that I stopped running out of disk space under
TravisCI ;-}

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.