[PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Beth Flanagan]

From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>

tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
buildstats to not collect diskio statistics when we encounter a case where 
the os.major/os.minor is not represented with an entry in /proc/diskstats.

Longer term solution to this is to:
- Identify all fs types that don't have representation in /proc/diskstats
- Gather meaningful iostats of these fs types if possible.

The following changes since commit 4a951e0433a99cd985515843f0a48fadc7050aca:

  Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700

Elizabeth Flanagan (1):
  [Yocto Bug 1700] Fix for buildstats on tmpfs

 meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
 1 files changed, 26 insertions(+), 11 deletions(-)

[PATCH 1/1] [Yocto Bug 1700] Fix for buildstats on tmpfs

[Beth Flanagan]

From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>

tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs
cause diskstats to choke. No device entry ends up in /proc/diskstats
for these fs types, which ends up causing the failure.

The short term solution is to exclude these fs types from diskstat
collection. Longer term we will want to see if we can collect
meaningful diskio for each of these, and other, use cases, but for
this cleans up Bug 1700.

Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
---
 meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
 1 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
index 55cbb3c..96c98d4 100644
--- a/meta/classes/buildstats.bbclass
+++ b/meta/classes/buildstats.bbclass
@@ -48,13 +48,24 @@ def set_device(e):
     # something like stick DL_DIR on a different partition and this would
     # throw stats gathering off. The same goes with SSTATE_DIR. However, let's
     # get the basics in here and work on the cornercases later. 
+    # A note. /proc/diskstats does not contain info on encryptfs, tmpfs, etc.
+    # If we end up hitting one of these fs, we'll just skip diskstats collection.
     ############################################################################
     device=os.stat(tmpdir)
     majordev=os.major(device.st_dev)
     minordev=os.minor(device.st_dev)
+    ############################################################################
+    # Bug 1700: 
+    # Because tmpfs/encryptfs/ramfs etc inserts no entry in /proc/diskstats
+    # we set rdev to NoLogicalDevice and search for it later. If we find NLD
+    # we do not collect diskstats as the method to collect meaningful statistics
+    # for these fs types requires a bit more research. 
+    ############################################################################
     for line in open("/proc/diskstats", "r"):
         if majordev == int(line.split()[0]) and minordev == int(line.split()[1]):
            rdev=line.split()[2]
+        else:
+           rdev="NoLogicalDevice"
     file = open(bb.data.getVar('DEVFILE', e.data, True), "w")
     file.write(rdev)
     file.close()
@@ -133,10 +144,11 @@ def write_task_data(status, logfile, dev, e):
     # For the best information, running things with BB_TOTAL_THREADS = "1"
     # would return accurate per task results.
     ############################################################################
-    diskdata = get_diskdata("__diskdata_task", dev, e.data)
-    if diskdata:
-        for key in sorted(diskdata.iterkeys()):
-            file.write(key + ": " + diskdata[key] + "\n")
+    if dev != "NoLogicalDevice":
+        diskdata = get_diskdata("__diskdata_task", dev, e.data)
+        if diskdata:
+            for key in sorted(diskdata.iterkeys()):
+                file.write(key + ": " + diskdata[key] + "\n")
     if status is "passed":
 	    file.write("Status: PASSED \n")
     else:
@@ -169,7 +181,8 @@ python run_buildstats () {
             bb.mkdirhier(bsdir)
         except:
             pass
-        set_diskdata("__diskdata_build", device, e.data)
+        if device != "NoLogicalDevice":
+            set_diskdata("__diskdata_build", device, e.data)
         set_timedata("__timedata_build", e.data)
         build_time = os.path.join(bsdir, "build_stats")
         # write start of build into build_time
@@ -185,7 +198,7 @@ python run_buildstats () {
                 
     elif isinstance(e, bb.event.BuildCompleted):
         bn = get_bn(e)
-        dev = get_device(e)
+        device = get_device(e)
         bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn)
         taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data))
         build_time = os.path.join(bsdir, "build_stats")
@@ -201,10 +214,11 @@ python run_buildstats () {
             file.write("Elapsed time: %0.2f seconds \n" % (time))
             if cpu:
                 file.write("CPU usage: %0.1f%% \n" % cpu)
-        diskio = get_diskdata("__diskdata_build", dev, e.data)
-        if diskio:
-            for key in sorted(diskio.iterkeys()):
-                file.write(key + ": " + diskio[key] + "\n")
+        if device != "NoLogicalDevice":
+            diskio = get_diskdata("__diskdata_build", device, e.data)
+            if diskio:
+                for key in sorted(diskio.iterkeys()):
+                    file.write(key + ": " + diskio[key] + "\n")
         file.close()
 
     if isinstance(e, bb.build.TaskStarted):
@@ -212,7 +226,8 @@ python run_buildstats () {
         device = get_device(e)
         bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn)
         taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data))
-        set_diskdata("__diskdata_task", device, e.data)
+        if device != "NoLogicalDevice":
+            set_diskdata("__diskdata_task", device, e.data)
         set_timedata("__timedata_task", e.data)
         try:
             bb.mkdirhier(taskdir)
-- 
1.7.1

Re: [PATCH 1/1] [Yocto Bug 1700] Fix for buildstats on tmpfs

[Saul Wold]

On 11/01/2011 11:41 PM, Beth Flanagan wrote:
> From: Elizabeth Flanagan<elizabeth.flanagan@intel.com>
>
> tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs
> cause diskstats to choke. No device entry ends up in /proc/diskstats
> for these fs types, which ends up causing the failure.
>
> The short term solution is to exclude these fs types from diskstat
> collection. Longer term we will want to see if we can collect
> meaningful diskio for each of these, and other, use cases, but for
> this cleans up Bug 1700.
>
> Signed-off-by: Elizabeth Flanagan<elizabeth.flanagan@intel.com>
> ---
>   meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
>   1 files changed, 26 insertions(+), 11 deletions(-)
>
> diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
> index 55cbb3c..96c98d4 100644
> --- a/meta/classes/buildstats.bbclass
> +++ b/meta/classes/buildstats.bbclass
> @@ -48,13 +48,24 @@ def set_device(e):
>       # something like stick DL_DIR on a different partition and this would
>       # throw stats gathering off. The same goes with SSTATE_DIR. However, let's
>       # get the basics in here and work on the cornercases later.
> +    # A note. /proc/diskstats does not contain info on encryptfs, tmpfs, etc.
> +    # If we end up hitting one of these fs, we'll just skip diskstats collection.
>       ############################################################################
>       device=os.stat(tmpdir)
>       majordev=os.major(device.st_dev)
>       minordev=os.minor(device.st_dev)
> +    ############################################################################
> +    # Bug 1700:
> +    # Because tmpfs/encryptfs/ramfs etc inserts no entry in /proc/diskstats
> +    # we set rdev to NoLogicalDevice and search for it later. If we find NLD
> +    # we do not collect diskstats as the method to collect meaningful statistics
> +    # for these fs types requires a bit more research.
> +    ############################################################################
>       for line in open("/proc/diskstats", "r"):
>           if majordev == int(line.split()[0]) and minordev == int(line.split()[1]):
>              rdev=line.split()[2]
> +        else:
> +           rdev="NoLogicalDevice"
>       file = open(bb.data.getVar('DEVFILE', e.data, True), "w")
>       file.write(rdev)
>       file.close()
> @@ -133,10 +144,11 @@ def write_task_data(status, logfile, dev, e):
>       # For the best information, running things with BB_TOTAL_THREADS = "1"
>       # would return accurate per task results.
>       ############################################################################
> -    diskdata = get_diskdata("__diskdata_task", dev, e.data)
> -    if diskdata:
> -        for key in sorted(diskdata.iterkeys()):
> -            file.write(key + ": " + diskdata[key] + "\n")
> +    if dev != "NoLogicalDevice":
> +        diskdata = get_diskdata("__diskdata_task", dev, e.data)
> +        if diskdata:
> +            for key in sorted(diskdata.iterkeys()):
> +                file.write(key + ": " + diskdata[key] + "\n")
>       if status is "passed":
>   	    file.write("Status: PASSED \n")
>       else:
> @@ -169,7 +181,8 @@ python run_buildstats () {
>               bb.mkdirhier(bsdir)
>           except:
>               pass
> -        set_diskdata("__diskdata_build", device, e.data)
> +        if device != "NoLogicalDevice":
> +            set_diskdata("__diskdata_build", device, e.data)
>           set_timedata("__timedata_build", e.data)
>           build_time = os.path.join(bsdir, "build_stats")
>           # write start of build into build_time
> @@ -185,7 +198,7 @@ python run_buildstats () {
>
>       elif isinstance(e, bb.event.BuildCompleted):
>           bn = get_bn(e)
> -        dev = get_device(e)
> +        device = get_device(e)
>           bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn)
>           taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data))
>           build_time = os.path.join(bsdir, "build_stats")
> @@ -201,10 +214,11 @@ python run_buildstats () {
>               file.write("Elapsed time: %0.2f seconds \n" % (time))
>               if cpu:
>                   file.write("CPU usage: %0.1f%% \n" % cpu)
> -        diskio = get_diskdata("__diskdata_build", dev, e.data)
> -        if diskio:
> -            for key in sorted(diskio.iterkeys()):
> -                file.write(key + ": " + diskio[key] + "\n")
> +        if device != "NoLogicalDevice":
> +            diskio = get_diskdata("__diskdata_build", device, e.data)
> +            if diskio:
> +                for key in sorted(diskio.iterkeys()):
> +                    file.write(key + ": " + diskio[key] + "\n")
>           file.close()
>
>       if isinstance(e, bb.build.TaskStarted):
> @@ -212,7 +226,8 @@ python run_buildstats () {
>           device = get_device(e)
>           bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn)
>           taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data))
> -        set_diskdata("__diskdata_task", device, e.data)
> +        if device != "NoLogicalDevice":
> +            set_diskdata("__diskdata_task", device, e.data)
>           set_timedata("__timedata_task", e.data)
>           try:
>               bb.mkdirhier(taskdir)

Merged into OE-Core with minor tweak to fix commit message formating

Thanks
	Sau!

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Wolfram Stering]

On 11/02/2011 07:41 AM, Beth Flanagan wrote:
> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>
> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
> buildstats to not collect diskio statistics when we encounter a case where 
> the os.major/os.minor is not represented with an entry in /proc/diskstats.

A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes.
The problem there is, that btrfs's stat() reports fake device ids that
cannot be found in /proc/diskstats.

> Longer term solution to this is to:
> - Identify all fs types that don't have representation in /proc/diskstats
> - Gather meaningful iostats of these fs types if possible.
>
> The following changes since commit 4a951e0433a99cd985515843f0a48fadc7050aca:
>
>   Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)
>
> are available in the git repository at:
>   git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700
>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700
>
> Elizabeth Flanagan (1):
>   [Yocto Bug 1700] Fix for buildstats on tmpfs
>
>  meta/classes/buildstats.bbclass |   37 ++++++++++++++++++++++++++-----------
>  1 files changed, 26 insertions(+), 11 deletions(-)
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


-- 
DI Wolfram Stering
 (Entwicklung)
HALE electronic GmbH
Eugen-Müller-Straße 18, 5020 Salzburg, Austria
 Tel: +43 (662) 439011 550
 Fax: +43 (662) 439011 9
http://www.hale.at/
Firmenbuchnummer: FN 66801m HG Salzburg



--
Scanned by MailScanner.

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Richard Purdie]

On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
> > From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
> >
> > tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
> > buildstats to not collect diskio statistics when we encounter a case where 
> > the os.major/os.minor is not represented with an entry in /proc/diskstats.
> 
> A similar issue exists for building on a btrfs partition.
> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
> The problem there is, that btrfs's stat() reports fake device ids that
> cannot be found in /proc/diskstats.

Did this patch help address that problem for you too?

Cheers,

Richard

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Wolfram Stering]

On 11/08/2011 03:12 PM, Richard Purdie wrote:
> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>
>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies 
>>> buildstats to not collect diskio statistics when we encounter a case where 
>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>> A similar issue exists for building on a btrfs partition.
>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>> The problem there is, that btrfs's stat() reports fake device ids that
>> cannot be found in /proc/diskstats.
> Did this patch help address that problem for you too?

I'll be able to check that tomorrow and report back.

regards,

- wolfi

> Cheers,
>
> Richard



-- 
Wolfram Stering
 (Entwicklung)
HALE electronic GmbH
Eugen-Müller-Straße 18, 5020 Salzburg, Austria
 Tel: +43 (662) 439011 550
 Fax: +43 (662) 439011 9
http://www.hale.at/
Firmenbuchnummer: FN 66801m HG Salzburg



--
Scanned by MailScanner.

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Flanagan, Elizabeth]

On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>
>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>> A similar issue exists for building on a btrfs partition.
>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>> The problem there is, that btrfs's stat() reports fake device ids that
>>> cannot be found in /proc/diskstats.
>> Did this patch help address that problem for you too?
>
> I'll be able to check that tomorrow and report back.

It should as the patch will just disable diskio collection if it
cannot find a valid device id in /proc/diskstats. Let me know if it
doesn't and I'll rework it and resubmit.

-b

>
> regards,
>
> - wolfi
>
>> Cheers,
>>
>> Richard
>
>
>
> --
> Wolfram Stering
>  (Entwicklung)
> HALE electronic GmbH
> Eugen-Müller-Straße 18, 5020 Salzburg, Austria
>  Tel: +43 (662) 439011 550
>  Fax: +43 (662) 439011 9
> http://www.hale.at/
> Firmenbuchnummer: FN 66801m HG Salzburg
>
>
>
> --
> Scanned by MailScanner.
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>



-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Wolfram Stering]

On 11/08/2011 04:57 PM, Flanagan, Elizabeth wrote:
> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>
>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>> A similar issue exists for building on a btrfs partition.
>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>> cannot be found in /proc/diskstats.
>>> Did this patch help address that problem for you too?
>> I'll be able to check that tomorrow and report back.
> It should as the patch will just disable diskio collection if it
> cannot find a valid device id in /proc/diskstats. Let me know if it
> doesn't and I'll rework it and resubmit.
>
> -b

Your patch fixes the buildstats issue for building on a btrfs volume as
well.
bitbake no longer hits the exception and diskstats are omitted.

Theoretically, this information would be available for btrfs, but it is
not discoverable
in the way buildstats currently collects the disk statistics. However,
this is a seaprate
issue, I think.

Thanks a lot for fixing this,

-wolfi

-- 
Wolfram Stering
 (Entwicklung)
HALE electronic GmbH
Eugen-Müller-Straße 18, 5020 Salzburg, Austria
 Tel: +43 (662) 439011 550
 Fax: +43 (662) 439011 9
http://www.hale.at/
Firmenbuchnummer: FN 66801m HG Salzburg



--
Scanned by MailScanner.

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]


Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:

> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>> 
>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>> A similar issue exists for building on a btrfs partition.
>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>> cannot be found in /proc/diskstats.
>>> Did this patch help address that problem for you too?
>> 
>> I'll be able to check that tomorrow and report back.
> 
> It should as the patch will just disable diskio collection if it
> cannot find a valid device id in /proc/diskstats. Let me know if it
> doesn't and I'll rework it and resubmit.

Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Flanagan, Elizabeth]

On Wed, Nov 9, 2011 at 4:00 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
> On 11/08/2011 04:57 PM, Flanagan, Elizabeth wrote:
>> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>>
>>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>>> A similar issue exists for building on a btrfs partition.
>>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>>> cannot be found in /proc/diskstats.
>>>> Did this patch help address that problem for you too?
>>> I'll be able to check that tomorrow and report back.
>> It should as the patch will just disable diskio collection if it
>> cannot find a valid device id in /proc/diskstats. Let me know if it
>> doesn't and I'll rework it and resubmit.
>>
>> -b
>
> Your patch fixes the buildstats issue for building on a btrfs volume as
> well.
> bitbake no longer hits the exception and diskstats are omitted.
>
> Theoretically, this information would be available for btrfs, but it is
> not discoverable
> in the way buildstats currently collects the disk statistics. However,
> this is a seaprate
> issue, I think.

Yes, and I'm open for suggestions on the best way to implement finding
disk io data for the myriad of filesystem types out there. :/ Or at
the very list  a way to pull diskstats from a trimmed down list that
includes:

ext2/3/4
btrfs
tmpfs
encryptfs....

I'm not sure performance data is even collected for for non-block
devices. dstat and iostat as far as I can tell can't collect data for
tmpfs.

Ideas?

-b

-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: [PATCH 0/1] [Yocto Bug 1700] Fix for buildstats diskio on non physical disks

[Flanagan, Elizabeth]

On Wed, Nov 9, 2011 at 4:13 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>
> Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:
>
>> On Tue, Nov 8, 2011 at 7:53 AM, Wolfram Stering <wolfram.stering@hale.at> wrote:
>>> On 11/08/2011 03:12 PM, Richard Purdie wrote:
>>>> On Tue, 2011-11-08 at 11:15 +0100, Wolfram Stering wrote:
>>>>> On 11/02/2011 07:41 AM, Beth Flanagan wrote:
>>>>>> From: Elizabeth Flanagan <elizabeth.flanagan@intel.com>
>>>>>>
>>>>>> tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies
>>>>>> buildstats to not collect diskio statistics when we encounter a case where
>>>>>> the os.major/os.minor is not represented with an entry in /proc/diskstats.
>>>>> A similar issue exists for building on a btrfs partition.
>>>>> I posted a message on Oct 28 concerning buildstats on btrfs volumes.
>>>>> The problem there is, that btrfs's stat() reports fake device ids that
>>>>> cannot be found in /proc/diskstats.
>>>> Did this patch help address that problem for you too?
>>>
>>> I'll be able to check that tomorrow and report back.
>>
>> It should as the patch will just disable diskio collection if it
>> cannot find a valid device id in /proc/diskstats. Let me know if it
>> doesn't and I'll rework it and resubmit.
>
> Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)
>
> regards,

Koen,

There is a patch to pybootchartgui that allows visualization. RP has
the patchset IIRC.  I'm going to be working on buildstats
visualization soon and I'm leaning towards using jquery/flot to create
a way to allow end users to generate via browser some comparative
graphing of buildstats against different builds/statistics. This way I
can set it up for the AB to compare all the builds we're doing there
and end users can just use it through a browser either locally or via
a server.

-b

>
> Koen
> -----BEGIN PGP SIGNATURE-----
>
> iEYEARECAAYFAk66bmcACgkQMkyGM64RGpHMCQCguHdgtS3SrIVV5+T1dOktTpAT
> FkQAnRZzZqwPJlGa3QlGGmxRrZnNtv/8
> =fEA3
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>



-- 
Elizabeth Flanagan
Yocto Project
Build and Release