From mboxrd@z Thu Jan 1 00:00:00 1970 From: Artem Bityutskiy Subject: FYI: bmaptool Date: Wed, 14 Aug 2013 14:48:18 +0300 Message-ID: <1376480898.2089.78.camel@sauron.fi.intel.com> Reply-To: dedekind1@gmail.com Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit To: Linux FS Maling List Return-path: Received: from mga02.intel.com ([134.134.136.20]:56317 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759892Ab3HNLnM (ORCPT ); Wed, 14 Aug 2013 07:43:12 -0400 Received: from [10.237.72.64] (sauron.fi.intel.com [10.237.72.64]) by linux.intel.com (Postfix) with ESMTP id DE8FB6A4007 for ; Wed, 14 Aug 2013 04:42:49 -0700 (PDT) Sender: linux-fsdevel-owner@vger.kernel.org List-ID: Hi, I'd like to introduce the bmap-tools project. I am sending this e-mail to this forum since here have people who may find it useful, or give good feed-back/criticism. This does not really have much to do with the kernel. Anyway, I am working with Tizen IVI and we use raw FS images there, and there has always be a problem that flashing the images to USB sticks takes very long time - we used dd for this. I addressed this issue with the bmaptool, which increased flashing speed by factor of 8 for Tizen IVI back at that time. In short, the idea is that raw images in Tizen IVI were very sparse, and there was a lot of 'gaps' in them (unmapped areas). Of course the sparseness information is lost as soon as the image is compressed and published. What I implemented is so-called bmap file, which I generate on the build server just after the raw image is generated. I use FIEMAP to record where the gaps are. And later this information is used to speed-up copying and flashing of the image - instead of copying all the zeroes, we just skip the area where there were gaps. This increases flashing speed dramatically for reasonably spars images. On top of that I implement various goodies in bmaptool which make it a lot more usable than plain 'dd', not only much faster: * it can read images from http/https/ssh/ftp urls, also via proxy * it can read compressed images and stream-uncompress them on-the-fly, supported formats are .gz, .xz, .bz2, .tgz, tar.{gz|xz|bz2} * it won't let you write to a mounted block device * it uses tricks like switching to noop scheduler for faster sequential writing to a block device, so it is faster than dd even without the bmap file * it reacts on ctrl-c almost immediately, unlike dd, because we fsync the block device after every 4MiB or something, so that ctrl-c won't need to flush huge amount of data before exiting (kernel forces full flush for bdevs on the last close) * check SHA1 of the data and bmap file (the plan is to switch to sha256 and support gpg signature verification) * some other things I guess :-) So, for example, in Tizen IVI we flash raw images like this: bmaptool copy http://download.tizen.org/.../image.raw.bz2 /dev/usb_stick This will read data from the URL, uncompress the data on-the-fly, and write to USB stick. It will also auto-discover the bmap file, and use it for ultra-fast writing (providing there is sparseness, which is often the case). It can also be use for copying large images around, not necessarily writing to block devices. FYI, this technique can also be used for Factory-flashing of images. Indeed, when have a product pipe-line, you want to populate them with data as fast as possible, and the fastest way to do this is to write directly to the block device and only those blocks which have data. Well, enough for first introduction. If someone is interested, see the docs here: https://source.tizen.org/documentation/articles/bmaptool It is a bit out-of-date, but not much, will be updated soon. Please, see the information about the project here as well: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/devel:/docs/README An example of a Tizen IVI image and the bmap file can be found here: http://linux-ftp.fi.intel.com/pub/mirrors/tizen/snapshots/tizen/ivi/latest/images/ivi-efi-i586/ Feed-back is welcome. I hope the project can be useful for others (it is GPL-2.0). And here is an example of a bmap file which I created today for a Fedora image (I sent a similar announcement to the Fedora community): 2147483648 4096 524288 176288 75b48dc596a5e92d7cc4935d8fcc3a91c2e48b0f 0-15 240-1967 8224-9583 9632-11599 11632-11775 33008-33023 33136-33567 34032-34095 34544-34591 34656-34671 35056-35087 35568-35615 35888-35903 36080-36255 36304-36511 36592-37071 37104-55551 62656-62687 62992-63023 64144-64175 65120-98559 98672-100975 101040-101615 101808-123759 123952-124015 124064-141151 141392-143327 143360-146319 146624-155391 155888-164095 164208-165983 166016-166047 166080-167103 167568-196143 196848-213247 221424-223311 223344-223391 223472-227375 229616-229631 295152-295167 295280-295295 -- Best Regards, Artem Bityutskiy