From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: by yocto-www.yoctoproject.org (Postfix, from userid 118) id 22683E00D5F; Tue, 16 Jan 2018 03:39:49 -0800 (PST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on yocto-www.yoctoproject.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-HAM-Report: * -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high * trust * [192.55.52.88 listed in list.dnswl.org] * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 7DEF5E00A9C for ; Tue, 16 Jan 2018 03:39:47 -0800 (PST) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2018 03:39:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,368,1511856000"; d="scan'208";a="10933694" Received: from linux.intel.com ([10.54.29.200]) by orsmga006.jf.intel.com with ESMTP; 16 Jan 2018 03:39:47 -0800 Received: from linux.intel.com (vmed.fi.intel.com [10.237.72.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by linux.intel.com (Postfix) with ESMTPS id 3A01E580328; Tue, 16 Jan 2018 03:39:46 -0800 (PST) Date: Tue, 16 Jan 2018 11:58:13 +0200 From: Ed Bartosh To: Martin Siegumfeldt Message-ID: <20180116095813.zpydtrfdpvwo3coe@linux.intel.com> References: MIME-Version: 1.0 In-Reply-To: Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: NeoMutt/20170421 (1.8.2) Cc: "yocto@yoctoproject.org" Subject: Re: wic- optimizing an image containing a large empty partition X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list Reply-To: ed.bartosh@linux.intel.com List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jan 2018 11:39:49 -0000 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Tue, Jan 16, 2018 at 09:22:07AM +0000, Martin Siegumfeldt wrote: > Hi, > > > I am trying to optimize an image having a fairly large empty partition with regards to file size - wks file: > > > martin@martin-Precision-5510:~/work/z7000-distro_wic/meta-z7000$ cat scripts/lib/wic/canned-wks/gs-boot-rootfs.wks > part --ondisk mmcblk --size 4 > part /boot --source bootimg-partition --ondisk mmcblk0 --fstype=vfat --label boot --active --align 4096 --size 64 --fsoptions ro > part / --source rootfs --ondisk mmcblk --fstype=ext4 --label root --align 4096 > part /mnt/data --ondisk mmcblk0 --fstype=ext4 --label data --size 1024 > > Now, generating the map file indicates that bmap-tools is capable of significantly reducing the data transfer: > > martin@martin-Precision-5510:~/work/z7000-distro_wic/build/tmp-glibc/deploy/images/zynq-soft-zedboard$ cat z7000-base-image-zynq-soft-zedboard.wic.bmap > > > > > > 1257452544 > > > 4096 > > > 306996 > > > 10765 > > > sha256 > > > d41d210e8efdf32597e566360a2f701e706f3b04bcf2418bb0c50a1f8a8d6c72 > > > > 0-6 > 2048-2347 > 23552-23620 > 23622-23688 > 23957-25600 > 25664-29696 > 29760-31744 > 32768-33792 > 33856-35389 > 37888 > 41984 > 44851-44917 > 44920-44921 > 44923-44925 > 44932-44933 > 53124-53130 > 77619-77621 > 143155-143157 > 175923-175924 > 208691-208693 > 274227-274229 > 306992-306995 > > > > Unfortunately, my HW does not integrate an SD card (that could have been flashed using bmap-tools) and we currently flash the soldered eMMC using dfu-util. In this case all 1.2 GB are transferred, which I consider fairly suboptimal compared to the above reported 42 MB 😞. > After briefly looking around at dfu-util and .dfu format description I'd suggest to convert wic images into .dfu using either information from .bmap or wic filemap API(wrapper around FIEMAP ioctl or the 'SEEK_HOLE / SEEK_DATA' features of the file seek syscall) and then use dfu-util to upload an image to the target media. > In the past we maintained a custom image class that simply skipped creating the empty ext4 partition and left it to be created upon the first boot. I could probably live with this approach, but I don't really see an easy way of achieveing this using wic - perhaps I missed it? However, if possible, my preference would be to generate the partition during the build and not at runtime. Can you elaborate on this a bit more? My guess is that you want to create unformatted partition, but I'm not sure I understand how that can help in this particular case. > > I see some history of sparse images related to wic, but the works appears reverted? > Can you point me out to the reverted commits? wic images are sparse. Here is an example of 94M sparse image with allocated size 16M: $ ls -lhs ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64-20180116091809.rootfs.wic 16M -rw-r--r-- 2 ed users 94M Jan 16 11:22 ./tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64-20180116091809.rootfs.wic -- Regards, Ed