[U-Boot] Fastboot behaviour with sparse images

[U-Boot] Fastboot behaviour with sparse images

[Maxime Ripard]

Hi,

I'm currently writing the support in U-Boot for NAND-backed devices
using fastboot [1], and that work derived a bit to supporting the
sparse images.

For "regular" images that are being stored, we expect a pair of
download and flash commands. Simple.

Things start to get a bit more complex with sparse images that have
been split because of a max-download-size lower than the actual image
size.

Here, from what I could gather from various random blog posts, the
fastboot client implementation and dumping a few USB sessions, the
client simply creates several download / flash pairs, always on the
same partition, without any way to distinct that from several
subsequent writes issued by the user.

So, I'm guessing that the expectation is that the bootloader
implementation should store the last offset it wrote to, and simple
resume from there if the partition names in the flash commands are the
same, which would prevent two subsequent write on the same partition
by any client. Am I right?

A related question is when should we erase the NAND partition? Only
when doing fastboot erase, or also when doing fastboot write (which,
combined with the issue raised above, would also mean that we don't
want to do an erase on the whole partition everytime there's a flash
command on it).

Thanks!
Maxime

1: http://lists.denx.de/pipermail/u-boot/2015-August/226053.html

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151012/5b7991e0/attachment.sig>

[U-Boot] Fastboot behaviour with sparse images

[Tom Rini]

On Mon, Oct 12, 2015 at 03:43:01PM +0200, Maxime Ripard wrote:
> Hi,
> 
> I'm currently writing the support in U-Boot for NAND-backed devices
> using fastboot [1], and that work derived a bit to supporting the
> sparse images.
> 
> For "regular" images that are being stored, we expect a pair of
> download and flash commands. Simple.
> 
> Things start to get a bit more complex with sparse images that have
> been split because of a max-download-size lower than the actual image
> size.
> 
> Here, from what I could gather from various random blog posts, the
> fastboot client implementation and dumping a few USB sessions, the
> client simply creates several download / flash pairs, always on the
> same partition, without any way to distinct that from several
> subsequent writes issued by the user.
> 
> So, I'm guessing that the expectation is that the bootloader
> implementation should store the last offset it wrote to, and simple
> resume from there if the partition names in the flash commands are the
> same, which would prevent two subsequent write on the same partition
> by any client. Am I right?
> 
> A related question is when should we erase the NAND partition? Only
> when doing fastboot erase, or also when doing fastboot write (which,
> combined with the issue raised above, would also mean that we don't
> want to do an erase on the whole partition everytime there's a flash
> command on it).

I think for this last question, some experimentation with the existing
tools might be required.  As there's no required explicit erase for MMC,
I think it might make sense to say you erase nand up front and then
write as anything else starts getting really tricky and we're just
second-guessing the user.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151021/410715c7/attachment.sig>

[U-Boot] Fastboot behaviour with sparse images

[Maxime Ripard]

Hi Tom,

On Wed, Oct 21, 2015 at 10:09:28AM -0400, Tom Rini wrote:
> On Mon, Oct 12, 2015 at 03:43:01PM +0200, Maxime Ripard wrote:
> > Hi,
> > 
> > I'm currently writing the support in U-Boot for NAND-backed devices
> > using fastboot [1], and that work derived a bit to supporting the
> > sparse images.
> > 
> > For "regular" images that are being stored, we expect a pair of
> > download and flash commands. Simple.
> > 
> > Things start to get a bit more complex with sparse images that have
> > been split because of a max-download-size lower than the actual image
> > size.
> > 
> > Here, from what I could gather from various random blog posts, the
> > fastboot client implementation and dumping a few USB sessions, the
> > client simply creates several download / flash pairs, always on the
> > same partition, without any way to distinct that from several
> > subsequent writes issued by the user.
> > 
> > So, I'm guessing that the expectation is that the bootloader
> > implementation should store the last offset it wrote to, and simple
> > resume from there if the partition names in the flash commands are the
> > same, which would prevent two subsequent write on the same partition
> > by any client. Am I right?
> > 
> > A related question is when should we erase the NAND partition? Only
> > when doing fastboot erase, or also when doing fastboot write (which,
> > combined with the issue raised above, would also mean that we don't
> > want to do an erase on the whole partition everytime there's a flash
> > command on it).
> 
> I think for this last question, some experimentation with the existing
> tools might be required.  As there's no required explicit erase for MMC,
> I think it might make sense to say you erase nand up front and then
> write as anything else starts getting really tricky and we're just
> second-guessing the user.

Actually, the only FS the fastboot tool seems to be doing it for the
moment are ext4 and F2FS. It can probably be extended to UBI and raw
partitions, but that won't fix the tools that are bundled by the
distros at the moment.

So I guess we can always erase it now using the session counter: if we
are writing the first chunk, erase the whole partition, if we're not,
then simply flash it at the previous offset.

How does it sound?
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151022/80b3135f/attachment.sig>

[U-Boot] Fastboot behaviour with sparse images

[Tom Rini]

On Thu, Oct 22, 2015 at 09:57:05AM +0200, Maxime Ripard wrote:
> Hi Tom,
> 
> On Wed, Oct 21, 2015 at 10:09:28AM -0400, Tom Rini wrote:
> > On Mon, Oct 12, 2015 at 03:43:01PM +0200, Maxime Ripard wrote:
> > > Hi,
> > > 
> > > I'm currently writing the support in U-Boot for NAND-backed devices
> > > using fastboot [1], and that work derived a bit to supporting the
> > > sparse images.
> > > 
> > > For "regular" images that are being stored, we expect a pair of
> > > download and flash commands. Simple.
> > > 
> > > Things start to get a bit more complex with sparse images that have
> > > been split because of a max-download-size lower than the actual image
> > > size.
> > > 
> > > Here, from what I could gather from various random blog posts, the
> > > fastboot client implementation and dumping a few USB sessions, the
> > > client simply creates several download / flash pairs, always on the
> > > same partition, without any way to distinct that from several
> > > subsequent writes issued by the user.
> > > 
> > > So, I'm guessing that the expectation is that the bootloader
> > > implementation should store the last offset it wrote to, and simple
> > > resume from there if the partition names in the flash commands are the
> > > same, which would prevent two subsequent write on the same partition
> > > by any client. Am I right?
> > > 
> > > A related question is when should we erase the NAND partition? Only
> > > when doing fastboot erase, or also when doing fastboot write (which,
> > > combined with the issue raised above, would also mean that we don't
> > > want to do an erase on the whole partition everytime there's a flash
> > > command on it).
> > 
> > I think for this last question, some experimentation with the existing
> > tools might be required.  As there's no required explicit erase for MMC,
> > I think it might make sense to say you erase nand up front and then
> > write as anything else starts getting really tricky and we're just
> > second-guessing the user.
> 
> Actually, the only FS the fastboot tool seems to be doing it for the
> moment are ext4 and F2FS. It can probably be extended to UBI and raw
> partitions, but that won't fix the tools that are bundled by the
> distros at the moment.
> 
> So I guess we can always erase it now using the session counter: if we
> are writing the first chunk, erase the whole partition, if we're not,
> then simply flash it at the previous offset.
> 
> How does it sound?

Sounds workable but testing with the existing tools will be the key and
the hard part here :(

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151022/aea21652/attachment.sig>

[U-Boot] Fastboot behaviour with sparse images

[Maxime Ripard]

On Thu, Oct 22, 2015 at 07:22:54AM -0400, Tom Rini wrote:
> On Thu, Oct 22, 2015 at 09:57:05AM +0200, Maxime Ripard wrote:
> > Hi Tom,
> > 
> > On Wed, Oct 21, 2015 at 10:09:28AM -0400, Tom Rini wrote:
> > > On Mon, Oct 12, 2015 at 03:43:01PM +0200, Maxime Ripard wrote:
> > > > Hi,
> > > > 
> > > > I'm currently writing the support in U-Boot for NAND-backed devices
> > > > using fastboot [1], and that work derived a bit to supporting the
> > > > sparse images.
> > > > 
> > > > For "regular" images that are being stored, we expect a pair of
> > > > download and flash commands. Simple.
> > > > 
> > > > Things start to get a bit more complex with sparse images that have
> > > > been split because of a max-download-size lower than the actual image
> > > > size.
> > > > 
> > > > Here, from what I could gather from various random blog posts, the
> > > > fastboot client implementation and dumping a few USB sessions, the
> > > > client simply creates several download / flash pairs, always on the
> > > > same partition, without any way to distinct that from several
> > > > subsequent writes issued by the user.
> > > > 
> > > > So, I'm guessing that the expectation is that the bootloader
> > > > implementation should store the last offset it wrote to, and simple
> > > > resume from there if the partition names in the flash commands are the
> > > > same, which would prevent two subsequent write on the same partition
> > > > by any client. Am I right?
> > > > 
> > > > A related question is when should we erase the NAND partition? Only
> > > > when doing fastboot erase, or also when doing fastboot write (which,
> > > > combined with the issue raised above, would also mean that we don't
> > > > want to do an erase on the whole partition everytime there's a flash
> > > > command on it).
> > > 
> > > I think for this last question, some experimentation with the existing
> > > tools might be required.  As there's no required explicit erase for MMC,
> > > I think it might make sense to say you erase nand up front and then
> > > write as anything else starts getting really tricky and we're just
> > > second-guessing the user.
> > 
> > Actually, the only FS the fastboot tool seems to be doing it for the
> > moment are ext4 and F2FS. It can probably be extended to UBI and raw
> > partitions, but that won't fix the tools that are bundled by the
> > distros at the moment.
> > 
> > So I guess we can always erase it now using the session counter: if we
> > are writing the first chunk, erase the whole partition, if we're not,
> > then simply flash it at the previous offset.
> > 
> > How does it sound?
> 
> Sounds workable but testing with the existing tools will be the key and
> the hard part here :(

Well, if we always erase when we write, the worst case scenario would
be one erase too many. It doesn't sound that bad, or hard.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151022/0215fabc/attachment.sig>

[U-Boot] Fastboot behaviour with sparse images

[Colin Cross]

On Monday, October 12, 2015 at 6:43:37 AM UTC-7, Maxime Ripard wrote:
>
> Hi, 
>
> I'm currently writing the support in U-Boot for NAND-backed devices 
> using fastboot [1], and that work derived a bit to supporting the 
> sparse images. 
>
> For "regular" images that are being stored, we expect a pair of 
> download and flash commands. Simple. 
>
> Things start to get a bit more complex with sparse images that have 
> been split because of a max-download-size lower than the actual image 
> size. 
>
> Here, from what I could gather from various random blog posts, the 
> fastboot client implementation and dumping a few USB sessions, the 
> client simply creates several download / flash pairs, always on the 
> same partition, without any way to distinct that from several 
> subsequent writes issued by the user. 
>
> So, I'm guessing that the expectation is that the bootloader 
> implementation should store the last offset it wrote to, and simple 
> resume from there if the partition names in the flash commands are the 
> same, which would prevent two subsequent write on the same partition 
> by any client. Am I right? 
>

No, each blob passed to the bootloader will begin with a sparse "skip" 
chunk that will seek to the correct place to resume writing.  The 
bootloader shouldn't need to store any metadata across commands.  Just read 
in the blob from the data command, then write it out using a port of the 
Apache-licensed libsparse during the flash command.
 

> A related question is when should we erase the NAND partition? Only 
> when doing fastboot erase, or also when doing fastboot write (which, 
> combined with the issue raised above, would also mean that we don't 
> want to do an erase on the whole partition everytime there's a flash 
> command on it).


Fastboot should send an erase command before every sequence of writes. 
 Erase the whole partition on the erase command, and don't erase anything 
on the flash command.
 

>  
>
Thanks! 
> Maxime 
>
> 1: http://lists.denx.de/pipermail/u-boot/2015-August/226053.html 
>
> -- 
> Maxime Ripard, Free Electrons 
> Embedded Linux, Kernel and Android engineering 
> http://free-electrons.com 
>

[U-Boot] Fastboot behaviour with sparse images

[Maxime Ripard]

Hi Colin,

On Tue, Oct 27, 2015 at 02:00:24PM -0700, Colin Cross wrote:
> On Monday, October 12, 2015 at 6:43:37 AM UTC-7, Maxime Ripard wrote:
> >
> > Hi, 
> >
> > I'm currently writing the support in U-Boot for NAND-backed devices 
> > using fastboot [1], and that work derived a bit to supporting the 
> > sparse images. 
> >
> > For "regular" images that are being stored, we expect a pair of 
> > download and flash commands. Simple. 
> >
> > Things start to get a bit more complex with sparse images that have 
> > been split because of a max-download-size lower than the actual image 
> > size. 
> >
> > Here, from what I could gather from various random blog posts, the 
> > fastboot client implementation and dumping a few USB sessions, the 
> > client simply creates several download / flash pairs, always on the 
> > same partition, without any way to distinct that from several 
> > subsequent writes issued by the user. 
> >
> > So, I'm guessing that the expectation is that the bootloader 
> > implementation should store the last offset it wrote to, and simple 
> > resume from there if the partition names in the flash commands are the 
> > same, which would prevent two subsequent write on the same partition 
> > by any client. Am I right? 
> >
> 
> No, each blob passed to the bootloader will begin with a sparse "skip" 
> chunk that will seek to the correct place to resume writing.  The 
> bootloader shouldn't need to store any metadata across commands.  Just read 
> in the blob from the data command, then write it out using a port of the 
> Apache-licensed libsparse during the flash command.

Oh, so that's how it works. Great. I guess however that you still need
to scan out the area you skip for bad blocks to account them in the
offset calculation as well then (when you're using NAND).

> > A related question is when should we erase the NAND partition? Only 
> > when doing fastboot erase, or also when doing fastboot write (which, 
> > combined with the issue raised above, would also mean that we don't 
> > want to do an erase on the whole partition everytime there's a flash 
> > command on it).
> 
> Fastboot should send an erase command before every sequence of writes. 
>  Erase the whole partition on the erase command, and don't erase anything 
> on the flash command.

Ack.

Thanks!
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20151104/033d9ae0/attachment.sig>