[PATCH] classes/rm_work: use the idle I/O scheduler class

[PATCH] classes/rm_work: use the idle I/O scheduler class

[Ross Burton]

As rm_work is just cleanup it shouldn't starve more important tasks such as
do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
scheduler class.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/rm_work.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 3ebf095..3e6c5ac 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -15,6 +15,9 @@
 # to try and reduce disk usage
 BB_SCHEDULER ?= "completion"
 
+# Run the rm_work task in the idle scheduling class
+BB_TASK_IONICE_LEVEL_task-rm_work = "3.0"
+
 RMWORK_ORIG_TASK := "${BB_DEFAULT_TASK}"
 BB_DEFAULT_TASK = "rm_work_all"
 
-- 
2.8.0.rc3

Re: [PATCH] classes/rm_work: use the idle I/O scheduler class

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

On Tue, Jun 14, 2016 at 8:18 AM, Ross Burton <ross.burton@intel.com> wrote:

> As rm_work is just cleanup it shouldn't starve more important tasks such as
> do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
> scheduler class.
>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
>

Nice.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 996 bytes --]

Re: [PATCH] classes/rm_work: use the idle I/O scheduler class

[Patrick Ohly]

On Tue, 2016-06-14 at 16:18 +0100, Ross Burton wrote:
> As rm_work is just cleanup it shouldn't starve more important tasks such as
> do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
> scheduler class.

Whether that's desirable depends a lot on the goals for rm_work: when I
tried to use it for TravisCI to get around some pretty tight disk space
constraints, I found that do_rm_work was often not scheduled early
enough because other tasks generating more files had higher priority.

Reducing the IO priority of do_rm_work may have the same effect: it
runs, but then instead of removing files, the system produces more of
them, thus increasing the risk of exhausting the disk space.

I suspect a lot of benchmarking will be needed to determine what really
works well and what doesn't.

I ended up writing a custom scheduler for running under TravisCI:
https://github.com/01org/meta-intel-iot-security/blob/master/scripts/rmwork.py

That orders do_rm_work before any other task and also orders all tasks
related to single recipe so that they run together, thus making it
possible to clean up after do_build sooner. As an additional tweak it
distinguishes between "compile" and "cleanup" tasks and can run
"cleanup" tasks when the normal scheduler wouldn't because
BB_NUMBER_THREADS is reached.

But it has the same problem: not enough benchmarking to really quantify
the effect. All I know is that I stopped running out of disk space under
TravisCI ;-}

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH] classes/rm_work: use the idle I/O scheduler class

[Andre McCurdy]

On Thu, Jun 16, 2016 at 2:04 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:
> On Tue, 2016-06-14 at 16:18 +0100, Ross Burton wrote:
>> As rm_work is just cleanup it shouldn't starve more important tasks such as
>> do_compile of I/O, so use BB_TASK_IONICE_LEVEL to run the task in the idle
>> scheduler class.
>
> Whether that's desirable depends a lot on the goals for rm_work: when I
> tried to use it for TravisCI to get around some pretty tight disk space
> constraints, I found that do_rm_work was often not scheduled early
> enough because other tasks generating more files had higher priority.
>
> Reducing the IO priority of do_rm_work may have the same effect: it
> runs, but then instead of removing files, the system produces more of
> them, thus increasing the risk of exhausting the disk space.
>
> I suspect a lot of benchmarking will be needed to determine what really
> works well and what doesn't.
>
> I ended up writing a custom scheduler for running under TravisCI:
> https://github.com/01org/meta-intel-iot-security/blob/master/scripts/rmwork.py

I like the idea of BB_NUMBER_COMPILE_THREADS. Would it be appropriate
to add support for that to the default scheduler?

> That orders do_rm_work before any other task and also orders all tasks
> related to single recipe so that they run together, thus making it
> possible to clean up after do_build sooner. As an additional tweak it
> distinguishes between "compile" and "cleanup" tasks and can run
> "cleanup" tasks when the normal scheduler wouldn't because
> BB_NUMBER_THREADS is reached.
>
> But it has the same problem: not enough benchmarking to really quantify
> the effect. All I know is that I stopped running out of disk space under
> TravisCI ;-}
>
> --
> Best Regards, Patrick Ohly
>
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
>
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core