[PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[Richard Purdie]

Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4, 
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest. 
The user can set this freely with normal privileges 

Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.

We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 948c395..22428a6 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
 
     bb.utils.mkdirhier(tempdir)
 
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index eb219c1..86a5328 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like psutil
+# however this gets us enough of what we need to bitbake for now without the
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS, who, ioprio)
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s" % rc)
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)

Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

On Sat, Oct 24, 2015 at 3:19 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index eb219c1..86a5328 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
>      result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
>      if result != 0:
>          raise PrCtlError('prctl failed with error code %s' % result)
> +
> +#
> +# Manually call the ioprio syscall. We could depend on other libs like
> psutil
> +# however this gets us enough of what we need to bitbake for now without
> the
> +# dependency
> +#
> +_unamearch = os.uname()[4]
> +IOPRIO_WHO_PROCESS = 1
> +IOPRIO_CLASS_SHIFT = 13
> +
> +def ioprio_set(who, cls, value):
> +    NR_ioprio_set = None
> +    if _unamearch == "x86_64":
> +      NR_ioprio_set = 251
> +    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
> +      NR_ioprio_set = 289
> +
> +    if NR_ioprio_set:
> +        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
> +        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS,
> who, ioprio)
> +        if rc != 0:
> +            raise ValueError("Unable to set ioprio, syscall returned %s"
> % rc)
>

Question, is the only way the syscall ran return non-zero the case where
the ioprio vaue is wrong? If it can fail for any other reason (e.g.
interrupted syscall), this probably isn't always a ValueError.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2234 bytes --]

Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 3975 bytes --]

Hi,

See
http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL
for the new variable description in the BitBake User Manual.  Note that I
did not include the future implementation information at the bottom of
Richard's explanation.  Also, there is a bit about the 'cfq scheduler' near
the end.  I did not know how to properly format this string (e.g. CFG
Scheduler).

Let me know of any changes or fixes.

Thanks,
Scott

On Sat, Oct 24, 2015 at 3:19 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the
> ioprio
> of tasks to be adjusted. This is in response to various qemu runtime
> timeouts
> which have been witnessed on the autobuilder, seemingly due to IO
> starvation (we
> already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to
> deal
> with certain 'random' failures we're seeing on the autobuilders in testing.
>
> The format of the data in the variable is BB_TASK_IONICE_LEVEL =
> "<class>.<prio>".
>
> For <class>, 2 is best effort (the default), 1 is real time and 3 is idle.
> You'd
> need superuser privileges to use realtime. The <prio> value is a default
> of 4,
> and can be set between 0 and 7 with 7 being lowest priority and 0 the
> highest.
> The user can set this freely with normal privileges
>
> Note that in order for this to take effect, you need the cfq scheduler
> selected
> for the backing block device.
>
> We could use nice wrapper functions for ioprio from modules like psutil
> however
> that would complicate bitbake dependencies. This version has some magic
> numbers
> but works on the main 32 and 64 bit x86 build architectures and can easily
> be
> extended if ever needed. When we move to python 3.x, we can likely replace
> this
> with standard calls.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index 948c395..22428a6 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
>          nice = int(nice) - curnice
>          newnice = os.nice(nice)
>          logger.debug(1, "Renice to %s " % newnice)
> +    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
> +    if ionice:
> +        try:
> +            cls, prio = ionice.split(".", 1)
> +            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
> +        except:
> +            bb.warn("Invalid ionice level %s" % ionice)
>
>      bb.utils.mkdirhier(tempdir)
>
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index eb219c1..86a5328 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
>      result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
>      if result != 0:
>          raise PrCtlError('prctl failed with error code %s' % result)
> +
> +#
> +# Manually call the ioprio syscall. We could depend on other libs like
> psutil
> +# however this gets us enough of what we need to bitbake for now without
> the
> +# dependency
> +#
> +_unamearch = os.uname()[4]
> +IOPRIO_WHO_PROCESS = 1
> +IOPRIO_CLASS_SHIFT = 13
> +
> +def ioprio_set(who, cls, value):
> +    NR_ioprio_set = None
> +    if _unamearch == "x86_64":
> +      NR_ioprio_set = 251
> +    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
> +      NR_ioprio_set = 289
> +
> +    if NR_ioprio_set:
> +        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
> +        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS,
> who, ioprio)
> +        if rc != 0:
> +            raise ValueError("Unable to set ioprio, syscall returned %s"
> % rc)
> +    else:
> +        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)
>
>
>

[-- Attachment #2: Type: text/html, Size: 4910 bytes --]

Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[Avery, Brian]

Hi,

It mostly appears as CFQ scheduler (Completely Fair Queuing).  Since we tell them to set it to cfq, it might be nice to tell them to do this by doing sudo sh -c “echo cfq > /sys/block/<device>/queu/scheduler; where device is the backing block device e.g. sda,sdb,…

-b

From: Scott Rifenbark <srifenbark@gmail.com<mailto:srifenbark@gmail.com>>
Date: Tuesday, October 27, 2015 at 6:34 AM
To: Richard Purdie <richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>>
Cc: bitbake-devel <bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>>, Brian Avery <brian.avery@intel.com<mailto:brian.avery@intel.com>>, "Witt, Randy E" <randy.e.witt@intel.com<mailto:randy.e.witt@intel.com>>, "Flanagan, Elizabeth" <elizabeth.flanagan@intel.com<mailto:elizabeth.flanagan@intel.com>>, "Graydon, Tracy" <tracy.graydon@intel.com<mailto:tracy.graydon@intel.com>>, Michael Halstead <michael@yoctoproject.org<mailto:michael@yoctoproject.org>>
Subject: Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

Hi,

See http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL for the new variable description in the BitBake User Manual.  Note that I did not include the future implementation information at the bottom of Richard's explanation.  Also, there is a bit about the 'cfq scheduler' near the end.  I did not know how to properly format this string (e.g. CFG Scheduler).

Let me know of any changes or fixes.

Thanks,
Scott

On Sat, Oct 24, 2015 at 3:19 AM, Richard Purdie <richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>> wrote:
Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio
of tasks to be adjusted. This is in response to various qemu runtime timeouts
which have been witnessed on the autobuilder, seemingly due to IO starvation (we
already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal
with certain 'random' failures we're seeing on the autobuilders in testing.

The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>".

For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd
need superuser privileges to use realtime. The <prio> value is a default of 4,
and can be set between 0 and 7 with 7 being lowest priority and 0 the highest.
The user can set this freely with normal privileges

Note that in order for this to take effect, you need the cfq scheduler selected
for the backing block device.

We could use nice wrapper functions for ioprio from modules like psutil however
that would complicate bitbake dependencies. This version has some magic numbers
but works on the main 32 and 64 bit x86 build architectures and can easily be
extended if ever needed. When we move to python 3.x, we can likely replace this
with standard calls.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org<mailto:richard.purdie@linuxfoundation.org>>

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 948c395..22428a6 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
         nice = int(nice) - curnice
         newnice = os.nice(nice)
         logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)

     bb.utils.mkdirhier(tempdir)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index eb219c1..86a5328 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
     result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
     if result != 0:
         raise PrCtlError('prctl failed with error code %s' % result)
+
+#
+# Manually call the ioprio syscall. We could depend on other libs like psutil
+# however this gets us enough of what we need to bitbake for now without the
+# dependency
+#
+_unamearch = os.uname()[4]
+IOPRIO_WHO_PROCESS = 1
+IOPRIO_CLASS_SHIFT = 13
+
+def ioprio_set(who, cls, value):
+    NR_ioprio_set = None
+    if _unamearch == "x86_64":
+      NR_ioprio_set = 251
+    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
+      NR_ioprio_set = 289
+
+    if NR_ioprio_set:
+        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
+        rc = cdll['libc.so.6'].syscall(NR_ioprio_set, IOPRIO_WHO_PROCESS, who, ioprio)
+        if rc != 0:
+            raise ValueError("Unable to set ioprio, syscall returned %s" % rc)
+    else:
+        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)

Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 5115 bytes --]

On Tue, Oct 27, 2015 at 7:59 AM, Avery, Brian <brian.avery@intel.com> wrote:

> Hi,
>
> It mostly appears as CFQ scheduler (Completely Fair Queuing).  Since we
> tell them to set it to cfq, it might be nice to tell them to do this by
> doing sudo sh -c “echo cfq > /sys/block/<device>/queu/scheduler; where
> device is the backing block device e.g. sda,sdb,…
>

Brian - I updated that note accordingly.  Is "queu" correct?  Or should it
be "queue"?

Thanks

>
> -b
>
> From: Scott Rifenbark <srifenbark@gmail.com>
> Date: Tuesday, October 27, 2015 at 6:34 AM
> To: Richard Purdie <richard.purdie@linuxfoundation.org>
> Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>, Brian Avery <
> brian.avery@intel.com>, "Witt, Randy E" <randy.e.witt@intel.com>,
> "Flanagan, Elizabeth" <elizabeth.flanagan@intel.com>, "Graydon, Tracy" <
> tracy.graydon@intel.com>, Michael Halstead <michael@yoctoproject.org>
> Subject: Re: [PATCH] build/utils: Add BB_TASK_IONICE_LEVEL support
>
> Hi,
>
> See
> http://www.yoctoproject.org/docs/2.0/bitbake-user-manual/bitbake-user-manual.html#var-BB_TASK_IONICE_LEVEL
> for the new variable description in the BitBake User Manual.  Note that I
> did not include the future implementation information at the bottom of
> Richard's explanation.  Also, there is a bit about the 'cfq scheduler' near
> the end.  I did not know how to properly format this string (e.g. CFG
> Scheduler).
>
> Let me know of any changes or fixes.
>
> Thanks,
> Scott
>
> On Sat, Oct 24, 2015 at 3:19 AM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows
>> the ioprio
>> of tasks to be adjusted. This is in response to various qemu runtime
>> timeouts
>> which have been witnessed on the autobuilder, seemingly due to IO
>> starvation (we
>> already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to
>> deal
>> with certain 'random' failures we're seeing on the autobuilders in
>> testing.
>>
>> The format of the data in the variable is BB_TASK_IONICE_LEVEL =
>> "<class>.<prio>".
>>
>> For <class>, 2 is best effort (the default), 1 is real time and 3 is
>> idle. You'd
>> need superuser privileges to use realtime. The <prio> value is a default
>> of 4,
>> and can be set between 0 and 7 with 7 being lowest priority and 0 the
>> highest.
>> The user can set this freely with normal privileges
>>
>> Note that in order for this to take effect, you need the cfq scheduler
>> selected
>> for the backing block device.
>>
>> We could use nice wrapper functions for ioprio from modules like psutil
>> however
>> that would complicate bitbake dependencies. This version has some magic
>> numbers
>> but works on the main 32 and 64 bit x86 build architectures and can
>> easily be
>> extended if ever needed. When we move to python 3.x, we can likely
>> replace this
>> with standard calls.
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
>> index 948c395..22428a6 100644
>> --- a/bitbake/lib/bb/build.py
>> +++ b/bitbake/lib/bb/build.py
>> @@ -413,6 +413,13 @@ def _exec_task(fn, task, d, quieterr):
>>          nice = int(nice) - curnice
>>          newnice = os.nice(nice)
>>          logger.debug(1, "Renice to %s " % newnice)
>> +    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
>> +    if ionice:
>> +        try:
>> +            cls, prio = ionice.split(".", 1)
>> +            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
>> +        except:
>> +            bb.warn("Invalid ionice level %s" % ionice)
>>
>>      bb.utils.mkdirhier(tempdir)
>>
>> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
>> index eb219c1..86a5328 100644
>> --- a/bitbake/lib/bb/utils.py
>> +++ b/bitbake/lib/bb/utils.py
>> @@ -1310,3 +1310,27 @@ def signal_on_parent_exit(signame):
>>      result = cdll['libc.so.6'].prctl(PR_SET_PDEATHSIG, signum)
>>      if result != 0:
>>          raise PrCtlError('prctl failed with error code %s' % result)
>> +
>> +#
>> +# Manually call the ioprio syscall. We could depend on other libs like
>> psutil
>> +# however this gets us enough of what we need to bitbake for now without
>> the
>> +# dependency
>> +#
>> +_unamearch = os.uname()[4]
>> +IOPRIO_WHO_PROCESS = 1
>> +IOPRIO_CLASS_SHIFT = 13
>> +
>> +def ioprio_set(who, cls, value):
>> +    NR_ioprio_set = None
>> +    if _unamearch == "x86_64":
>> +      NR_ioprio_set = 251
>> +    elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
>> +      NR_ioprio_set = 289
>> +
>> +    if NR_ioprio_set:
>> +        ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
>> +        rc = cdll['libc.so.6'].syscall(NR_ioprio_set,
>> IOPRIO_WHO_PROCESS, who, ioprio)
>> +        if rc != 0:
>> +            raise ValueError("Unable to set ioprio, syscall returned %s"
>> % rc)
>> +    else:
>> +        bb.warn("Unable to set IO Prio for arch %s" % _unamearch)
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 7914 bytes --]