[PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[Joshua Lock]

The individual partitions created by wic are sparse but without
this change the assembled image is written as one (potentially
very) large file.

Preserve sparseness in the assembled image by passing the sparse
conversion symbol.

Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
 scripts/lib/wic/utils/partitionedfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
index 5a103bb..3e2b420 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -340,7 +340,7 @@ class Image(object):
             source = part['source_file']
             if source:
                 # install source_file contents into a partition
-                cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc" % \
+                cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc,sparse" % \
                       (source, image_file, self.sector_size,
                        part['start'], part['size'])
                 exec_cmd(cmd)
-- 
2.5.0

Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[Ed Bartosh]

Hi Joshua,

Thank you for the patch! I like the change. It's a first step towards
supporting bmaptool, which is in my TODO list.

Would you be willing to write or modify wic test case to test this?

On Wed, Dec 16, 2015 at 04:33:13PM +0000, Joshua Lock wrote:
> The individual partitions created by wic are sparse but without
> this change the assembled image is written as one (potentially
> very) large file.
> 
> Preserve sparseness in the assembled image by passing the sparse
> conversion symbol.
> 
> Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
> ---
>  scripts/lib/wic/utils/partitionedfs.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
> index 5a103bb..3e2b420 100644
> --- a/scripts/lib/wic/utils/partitionedfs.py
> +++ b/scripts/lib/wic/utils/partitionedfs.py
> @@ -340,7 +340,7 @@ class Image(object):
>              source = part['source_file']
>              if source:
>                  # install source_file contents into a partition
> -                cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc" % \
> +                cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc,sparse" % \
>                        (source, image_file, self.sector_size,
>                         part['start'], part['size'])
>                  exec_cmd(cmd)
> -- 
> 2.5.0
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
--
Regards,
Ed

Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[Joshua Lock]

Hi Ed,

On 17/12/15 12:21, Ed Bartosh wrote:
> Thank you for the patch! I like the change. It's a first step towards
> supporting bmaptool, which is in my TODO list.

No problem, I've been playing with bmaptool myself and that's what 
prompted the change.

> Would you be willing to write or modify wic test case to test this?

I'd be willing to do that, is there any documentation on the development 
of test cases?

I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with 
@testcase decorators but I'm not familiar with how this should all fit 
together.

Regards,

Joshua

Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On 17 December 2015 at 13:19, Joshua Lock <joshua.lock@collabora.co.uk>
wrote:

> I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with
> @testcase decorators but I'm not familiar with how this should all fit
> together.
>

@testcase is just to map from test functions to testopia entries.  This
doesn't (yet) have a testopia entry, so you can leave it.

Apart from that it's standard Python unittest, so find a test that's
somewhat similar and copy it.

Ross

[-- Attachment #2: Type: text/html, Size: 971 bytes --]

Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files

[Ed Bartosh]

On Thu, Dec 17, 2015 at 01:19:47PM +0000, Joshua Lock wrote:
> Hi Ed,
> 
> On 17/12/15 12:21, Ed Bartosh wrote:
> >Thank you for the patch! I like the change. It's a first step towards
> >supporting bmaptool, which is in my TODO list.
> 
> No problem, I've been playing with bmaptool myself and that's what
> prompted the change.
> 
> >Would you be willing to write or modify wic test case to test this?
> 
> I'd be willing to do that, is there any documentation on the
> development of test cases?
You can find short info here:
https://wiki.yoctoproject.org/wiki/Oe-selftest

And here https://docs.python.org/2/library/unittest.html as oe-selftest
test cases are based on Python unittest.

> I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with
> @testcase decorators but I'm not familiar with how this should all
> fit together.
You've spotted it right. All wic testcases are in that file.
You can run them this way:  oe-selftest --run-tests wic

As Ross already pointed out you can omit @testcase decorator if you add
new test. QA team will add it later when they need it.

--
Regards,
Ed

[PATCH] oeqa/selftest/wic: add test case for sparse images

[Joshua Lock]

Add a testcase to build a directdisk image and check that the
used disk size is less than the apparent size, as wic now
assembles images as sparse files.

Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
 meta/lib/oeqa/selftest/wic.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 55f6f82..e85363c 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -264,3 +264,15 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_sparseness(self):
+        """Test that assembled images are sparse; apparent size > disk usage"""
+        self.assertEqual(0, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal").status)
+        images = glob(self.resultdir + "directdisk-*.direct")
+        self.assertEqual(1, len(images))
+
+        path = images.pop()
+        used = int(runCmd("du %s | awk '{print $1}'" % path).output)
+        apparent = int(runCmd("du --apparent-size %s | awk '{print $1}'" % path).output)
+        self.assertLess(used, apparent)
-- 
2.5.0

Re: [PATCH] oeqa/selftest/wic: add test case for sparse images

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

On 17 December 2015 at 16:09, Joshua Lock <joshua.lock@collabora.co.uk>
wrote:

> +        used = int(runCmd("du %s | awk '{print $1}'" % path).output)
> +        apparent = int(runCmd("du --apparent-size %s | awk '{print $1}'"
> % path).output)
>

Surely os.stat() can do this instead?

Ross

[-- Attachment #2: Type: text/html, Size: 759 bytes --]

Re: [PATCH] oeqa/selftest/wic: add test case for sparse images

[Joshua Lock]

On 17/12/15 16:21, Burton, Ross wrote:
>
> On 17 December 2015 at 16:09, Joshua Lock <joshua.lock@collabora.co.uk
> <mailto:joshua.lock@collabora.co.uk>> wrote:
>
>     +        used = int(runCmd("du %s | awk '{print $1}'" % path).output)
>     +        apparent = int(runCmd("du --apparent-size %s | awk '{print
>     $1}'" % path).output)
>
>
> Surely os.stat() can do this instead?

Of course it can — st_blocks is the "number of 512-byte blocks allocated 
for file" we can multiply st_blocks by 512 to get the on-disk size 
(whilst st_size is the apparent size).

https://docs.python.org/2/library/os.html#os.stat

I'll create a v2.

Joshua

[[PATCH v2] oeqa/selftest/wic: add test case for sparse images

[Joshua Lock]

Add a testcase to build a directdisk image and check that the
used disk size is less than the apparent size, as wic now
assembles images as sparse files.

Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
 meta/lib/oeqa/selftest/wic.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 55f6f82..9050d28 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -264,3 +264,16 @@ class Wic(oeSelfTest):
         self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
                                    % image).status)
         self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+    def test_sparseness(self):
+        """Test that assembled images are sparse; apparent size > disk usage"""
+        self.assertEqual(0, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal").status)
+        images = glob(self.resultdir + "directdisk-*.direct")
+        self.assertEqual(1, len(images))
+
+        imagestat = os.stat(images.pop())
+        # st_blocks is the "number of 512-byte blocks allocated for file"
+        used = imagestat.st_blocks*512
+        apparent = imagestat.st_size
+        self.assertLess(used, apparent)
-- 
2.5.0

Re: [[PATCH v2] oeqa/selftest/wic: add test case for sparse images

[Ed Bartosh]

Hi Joshua,

Thank you for the patch!

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>

On Thu, Dec 17, 2015 at 07:43:41PM +0000, Joshua Lock wrote:
> Add a testcase to build a directdisk image and check that the
> used disk size is less than the apparent size, as wic now
> assembles images as sparse files.
> 
> Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
> ---
>  meta/lib/oeqa/selftest/wic.py | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index 55f6f82..9050d28 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -264,3 +264,16 @@ class Wic(oeSelfTest):
>          self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
>                                     % image).status)
>          self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> +    def test_sparseness(self):
> +        """Test that assembled images are sparse; apparent size > disk usage"""
> +        self.assertEqual(0, runCmd("wic create directdisk "
> +                                   "--image-name core-image-minimal").status)
> +        images = glob(self.resultdir + "directdisk-*.direct")
> +        self.assertEqual(1, len(images))
> +
> +        imagestat = os.stat(images.pop())
> +        # st_blocks is the "number of 512-byte blocks allocated for file"
> +        used = imagestat.st_blocks*512
> +        apparent = imagestat.st_size
> +        self.assertLess(used, apparent)

--
Regards,
Ed