[U-Boot] dfu: dfu and UBI Volumes

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello,

just digging in DFU support in U-Boot for an upcoming board support
based on an AM335x. This board support uses for example a rootfs in
an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs
mtd partition with a nand erase and write the new image using
dfu_nand.c ... which calls in the end nand_write ... which is ...
lets say ... not the prefered way on an UBI volume ...

How to solve this? Any ideas?

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi Heiko,

On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:

> Hello,
> 
> just digging in DFU support in U-Boot for an upcoming board support
> based on an AM335x. This board support uses for example a rootfs in
> an UBI Volume on a NAND flash, and this should be updated with dfu ...
> 
> How To do this? Current state on this board is to erase the rootfs
> mtd partition with a nand erase and write the new image using
> dfu_nand.c ... which calls in the end nand_write ... which is ...
> lets say ... not the prefered way on an UBI volume ...
> 
> How to solve this? Any ideas?
> 

Well, what would you like ideally to do? Why is nand_write not ideal for
a UBI volume.

Note that dfu will skip over the bad blocks... 


> bye,
> Heiko
> -- 
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
> Hi Heiko,
> 
> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
> 
> > Hello,
> > 
> > just digging in DFU support in U-Boot for an upcoming board support
> > based on an AM335x. This board support uses for example a rootfs in
> > an UBI Volume on a NAND flash, and this should be updated with dfu ...
> > 
> > How To do this? Current state on this board is to erase the rootfs
> > mtd partition with a nand erase and write the new image using
> > dfu_nand.c ... which calls in the end nand_write ... which is ...
> > lets say ... not the prefered way on an UBI volume ...
> > 
> > How to solve this? Any ideas?
> 
> Well, what would you like ideally to do? Why is nand_write not ideal for
> a UBI volume.
> 
> Note that dfu will skip over the bad blocks... 

Presumably because they want to replace say ubi0:rootfs (and leave
ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write
in a new ubi container of everything.

I would suggest that, so long as our existing UBI infrastructure allows
this, you add a new method, dfu_ubi which takes care of programming
things.  This shouldn't be too bad to write as I've heard the existing
infrastucture was easily expanded for SPI (and patches are pending a
little more clean up prior to posting).

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130524/5bd80b11/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:
> On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
>> Hi Heiko,
>>
>> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
>>
>>> Hello,
>>>
>>> just digging in DFU support in U-Boot for an upcoming board support
>>> based on an AM335x. This board support uses for example a rootfs in
>>> an UBI Volume on a NAND flash, and this should be updated with dfu ...
>>>
>>> How To do this? Current state on this board is to erase the rootfs
>>> mtd partition with a nand erase and write the new image using
>>> dfu_nand.c ... which calls in the end nand_write ... which is ...
>>> lets say ... not the prefered way on an UBI volume ...
>>>
>>> How to solve this? Any ideas?
>>
>> Well, what would you like ideally to do? Why is nand_write not ideal for
>> a UBI volume.
>>
>> Note that dfu will skip over the bad blocks... 
> 
> Presumably because they want to replace say ubi0:rootfs (and leave
> ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write
> in a new ubi container of everything.
> 
> I would suggest that, so long as our existing UBI infrastructure allows
> this, you add a new method, dfu_ubi which takes care of programming
> things.  This shouldn't be too bad to write as I've heard the existing
> infrastucture was easily expanded for SPI (and patches are pending a
> little more clean up prior to posting).

This sounds easy ... but they have also raw nand partitions, for
example spl partitions on one nand flash ... for which dfu_nand.c
fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c
at the same time?

I would say no, if I understand the code right:
- starting dfu with "dfu interface={nand or mmc} ..."

and after that, you can not switch anymore between nand or mmc, right?

You must press Ctrl-C to end dfu and start it again for switching
to another interface ...

Or should I look to add ubi support to drivers/dfu/dfu_nand.c?

Something like:
Add in dfu_fill_entity_nand a "else if (!strcmp(st, "ubipart")) {"
(Did not looked deeper in this, if this is a possible way... and
 thinking about it ... it is not the correct way, ubi should be
 seperate, because it maybe runs also on nor flash ...)

So, the best way would be, to switch in "dfu nand" mode between
subinterfaces ... like "raw", "part", "ubi", ...

Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout
for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...

After all ... should we add a DFU_UBI and add this to
drivers/dfu/dfu_nand.c?

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Lukasz Majewski]

Hi Heiko,

> Hello Tom,
> 
> Am 24.05.2013 19:12, schrieb Tom Rini:
> > On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
> >> Hi Heiko,
> >>
> >> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
> >>
> >>> Hello,
> >>>
> >>> just digging in DFU support in U-Boot for an upcoming board
> >>> support based on an AM335x. This board support uses for example a
> >>> rootfs in an UBI Volume on a NAND flash, and this should be
> >>> updated with dfu ...
> >>>
> >>> How To do this? Current state on this board is to erase the rootfs
> >>> mtd partition with a nand erase and write the new image using
> >>> dfu_nand.c ... which calls in the end nand_write ... which is ...
> >>> lets say ... not the prefered way on an UBI volume ...
> >>>
> >>> How to solve this? Any ideas?
> >>
> >> Well, what would you like ideally to do? Why is nand_write not
> >> ideal for a UBI volume.
> >>
> >> Note that dfu will skip over the bad blocks... 
> > 
> > Presumably because they want to replace say ubi0:rootfs (and leave
> > ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than
> > write in a new ubi container of everything.
> > 
> > I would suggest that, so long as our existing UBI infrastructure
> > allows this, you add a new method, dfu_ubi which takes care of
> > programming things.  This shouldn't be too bad to write as I've
> > heard the existing infrastucture was easily expanded for SPI (and
> > patches are pending a little more clean up prior to posting).
> 
> This sounds easy ... but they have also raw nand partitions, for
> example spl partitions on one nand flash ... for which dfu_nand.c
> fits perfectly ... is it possible to use dfu_nand.c and another
> dfu_xxx.c at the same time?

I'm not so familiar with nand devices handling, but in my opinion we
shall create dfu_ubi.c file. 

I think that, nand part of dfu handling shall be separated from ubi,
even if UBI itself is layed on nand.

However, Tom and Pantelis shall give their opinion, since they spent a
lot of time on forcing dfu_nand.c to work.


> 
> I would say no, if I understand the code right:
> - starting dfu with "dfu interface={nand or mmc} ..."
> 
> and after that, you can not switch anymore between nand or mmc, right?
> 
> You must press Ctrl-C to end dfu and start it again for switching
> to another interface ...
> 
> Or should I look to add ubi support to drivers/dfu/dfu_nand.c?
> 
> Something like:
> Add in dfu_fill_entity_nand a "else if (!strcmp(st, "ubipart")) {"
> (Did not looked deeper in this, if this is a possible way... and
>  thinking about it ... it is not the correct way, ubi should be
>  seperate, because it maybe runs also on nor flash ...)
> 
> So, the best way would be, to switch in "dfu nand" mode between
> subinterfaces ... like "raw", "part", "ubi", ...
> 
> Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout
> for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...
> 
> After all ... should we add a DFU_UBI and add this to
> drivers/dfu/dfu_nand.c?
> 
> bye,
> Heiko



-- 
Best regards,

Lukasz Majewski

Samsung R&D Poland (SRPOL) | Linux Platform Group

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Lukasz,

Am 27.05.2013 09:02, schrieb Lukasz Majewski:
> Hi Heiko,
> 
>> Hello Tom,
>>
>> Am 24.05.2013 19:12, schrieb Tom Rini:
>>> On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
>>>> Hi Heiko,
>>>>
>>>> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> just digging in DFU support in U-Boot for an upcoming board
>>>>> support based on an AM335x. This board support uses for example a
>>>>> rootfs in an UBI Volume on a NAND flash, and this should be
>>>>> updated with dfu ...
>>>>>
>>>>> How To do this? Current state on this board is to erase the rootfs
>>>>> mtd partition with a nand erase and write the new image using
>>>>> dfu_nand.c ... which calls in the end nand_write ... which is ...
>>>>> lets say ... not the prefered way on an UBI volume ...
>>>>>
>>>>> How to solve this? Any ideas?
>>>>
>>>> Well, what would you like ideally to do? Why is nand_write not
>>>> ideal for a UBI volume.
>>>>
>>>> Note that dfu will skip over the bad blocks... 
>>>
>>> Presumably because they want to replace say ubi0:rootfs (and leave
>>> ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than
>>> write in a new ubi container of everything.
>>>
>>> I would suggest that, so long as our existing UBI infrastructure
>>> allows this, you add a new method, dfu_ubi which takes care of
>>> programming things.  This shouldn't be too bad to write as I've
>>> heard the existing infrastucture was easily expanded for SPI (and
>>> patches are pending a little more clean up prior to posting).
>>
>> This sounds easy ... but they have also raw nand partitions, for
>> example spl partitions on one nand flash ... for which dfu_nand.c
>> fits perfectly ... is it possible to use dfu_nand.c and another
>> dfu_xxx.c at the same time?
> 
> I'm not so familiar with nand devices handling, but in my opinion we
> shall create dfu_ubi.c file. 
> 
> I think that, nand part of dfu handling shall be separated from ubi,
> even if UBI itself is layed on nand.

Yes, I tend also to this .. but not as a separte "dfu interface .."
"dfu nand .." should stay, and each partition should know, if it
is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but
this should not be hardcoded in every dfu_xxx.c, instead it should
be something like a subinterface ... if it is possible.

> However, Tom and Pantelis shall give their opinion, since they spent a
> lot of time on forcing dfu_nand.c to work.

Yep, that would be great, as I am a dfu beginner ;-)

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi Guys,

On May 27, 2013, at 10:28 AM, Heiko Schocher wrote:

> Hello Lukasz,
> 
> Am 27.05.2013 09:02, schrieb Lukasz Majewski:
>> Hi Heiko,
>> 
>>> Hello Tom,
>>> 
>>> Am 24.05.2013 19:12, schrieb Tom Rini:
>>>> On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
>>>>> Hi Heiko,
>>>>> 
>>>>> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> just digging in DFU support in U-Boot for an upcoming board
>>>>>> support based on an AM335x. This board support uses for example a
>>>>>> rootfs in an UBI Volume on a NAND flash, and this should be
>>>>>> updated with dfu ...
>>>>>> 
>>>>>> How To do this? Current state on this board is to erase the rootfs
>>>>>> mtd partition with a nand erase and write the new image using
>>>>>> dfu_nand.c ... which calls in the end nand_write ... which is ...
>>>>>> lets say ... not the prefered way on an UBI volume ...
>>>>>> 
>>>>>> How to solve this? Any ideas?
>>>>> 
>>>>> Well, what would you like ideally to do? Why is nand_write not
>>>>> ideal for a UBI volume.
>>>>> 
>>>>> Note that dfu will skip over the bad blocks... 
>>>> 
>>>> Presumably because they want to replace say ubi0:rootfs (and leave
>>>> ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than
>>>> write in a new ubi container of everything.
>>>> 
>>>> I would suggest that, so long as our existing UBI infrastructure
>>>> allows this, you add a new method, dfu_ubi which takes care of
>>>> programming things.  This shouldn't be too bad to write as I've
>>>> heard the existing infrastucture was easily expanded for SPI (and
>>>> patches are pending a little more clean up prior to posting).
>>> 
>>> This sounds easy ... but they have also raw nand partitions, for
>>> example spl partitions on one nand flash ... for which dfu_nand.c
>>> fits perfectly ... is it possible to use dfu_nand.c and another
>>> dfu_xxx.c at the same time?
>> 
>> I'm not so familiar with nand devices handling, but in my opinion we
>> shall create dfu_ubi.c file. 
>> 
>> I think that, nand part of dfu handling shall be separated from ubi,
>> even if UBI itself is layed on nand.
> 
> Yes, I tend also to this .. but not as a separte "dfu interface .."
> "dfu nand .." should stay, and each partition should know, if it
> is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but
> this should not be hardcoded in every dfu_xxx.c, instead it should
> be something like a subinterface ... if it is possible.
> 

I'm not completely up to speed with UBI, but dfu_ubi seems to be the way
to go for me.

Looks like it's simple enough; erase (but don't step over the wear counters)
, write (but skip over the wear counters).

I don't know how smart you have to be with UBI version; be very careful
when the binary format of UBI changes.


>> However, Tom and Pantelis shall give their opinion, since they spent a
>> lot of time on forcing dfu_nand.c to work.
> 
> Yep, that would be great, as I am a dfu beginner ;-)
> 
> bye,
> Heiko
> -- 
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello,

Am 27.05.2013 09:35, schrieb Pantelis Antoniou:
> Hi Guys,
> 
> On May 27, 2013, at 10:28 AM, Heiko Schocher wrote:
> 
>> Hello Lukasz,
>>
>> Am 27.05.2013 09:02, schrieb Lukasz Majewski:
>>> Hi Heiko,
>>>
>>>> Hello Tom,
>>>>
>>>> Am 24.05.2013 19:12, schrieb Tom Rini:
>>>>> On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
>>>>>> Hi Heiko,
>>>>>>
>>>>>> On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> just digging in DFU support in U-Boot for an upcoming board
>>>>>>> support based on an AM335x. This board support uses for example a
>>>>>>> rootfs in an UBI Volume on a NAND flash, and this should be
>>>>>>> updated with dfu ...
>>>>>>>
>>>>>>> How To do this? Current state on this board is to erase the rootfs
>>>>>>> mtd partition with a nand erase and write the new image using
>>>>>>> dfu_nand.c ... which calls in the end nand_write ... which is ...
>>>>>>> lets say ... not the prefered way on an UBI volume ...
>>>>>>>
>>>>>>> How to solve this? Any ideas?
>>>>>>
>>>>>> Well, what would you like ideally to do? Why is nand_write not
>>>>>> ideal for a UBI volume.
>>>>>>
>>>>>> Note that dfu will skip over the bad blocks... 
>>>>>
>>>>> Presumably because they want to replace say ubi0:rootfs (and leave
>>>>> ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than
>>>>> write in a new ubi container of everything.
>>>>>
>>>>> I would suggest that, so long as our existing UBI infrastructure
>>>>> allows this, you add a new method, dfu_ubi which takes care of
>>>>> programming things.  This shouldn't be too bad to write as I've
>>>>> heard the existing infrastucture was easily expanded for SPI (and
>>>>> patches are pending a little more clean up prior to posting).
>>>>
>>>> This sounds easy ... but they have also raw nand partitions, for
>>>> example spl partitions on one nand flash ... for which dfu_nand.c
>>>> fits perfectly ... is it possible to use dfu_nand.c and another
>>>> dfu_xxx.c at the same time?
>>>
>>> I'm not so familiar with nand devices handling, but in my opinion we
>>> shall create dfu_ubi.c file. 
>>>
>>> I think that, nand part of dfu handling shall be separated from ubi,
>>> even if UBI itself is layed on nand.
>>
>> Yes, I tend also to this .. but not as a separte "dfu interface .."
>> "dfu nand .." should stay, and each partition should know, if it
>> is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but
>> this should not be hardcoded in every dfu_xxx.c, instead it should
>> be something like a subinterface ... if it is possible.
>>
> 
> I'm not completely up to speed with UBI, but dfu_ubi seems to be the way
> to go for me.

But how to handle a raw nand partition and a ubi partition on one
nand?

If ubi is a new dfu interface, somebody must start dfu on u-boot
with "dfu nand .." or "dfu ubi .." dependent on which partition
has to be updated ... before using dfu-util on the host side ...
and start dfu-util for the correct partition...

This seems not really userfriendly to me ... if I have to use the
u-boot shell, why using dfu-util on a host pc instead executing tftp
and update my images within u-boot?

Is ubi really a "interface" as nand or mmc ... ?

> Looks like it's simple enough; erase (but don't step over the wear counters)
> , write (but skip over the wear counters).

Yep, or load the complete image in ram, and write it with "ubi write ..."

> I don't know how smart you have to be with UBI version; be very careful
> when the binary format of UBI changes.

Thanks!

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Heiko Schocher,

In message <51A30F34.7030603@denx.de> you wrote:
> 
> But how to handle a raw nand partition and a ubi partition on one
> nand?
> 
> If ubi is a new dfu interface, somebody must start dfu on u-boot
> with "dfu nand .." or "dfu ubi .." dependent on which partition
> has to be updated ... before using dfu-util on the host side ...
> and start dfu-util for the correct partition...
> 
> This seems not really userfriendly to me ... if I have to use the

Indeed, this makes no sense and breaks the whole concept of DFU to be
able to download a sinlge firmware image with one, simple command.

> Is ubi really a "interface" as nand or mmc ... ?

No, it is not.  It could be considered a "partition type" at best.

> > Looks like it's simple enough; erase (but don't step over the wear counters)
> > , write (but skip over the wear counters).
> 
> Yep, or load the complete image in ram, and write it with "ubi write ..."

Not "or".  When dealing with UBI volumes, then "ubi write" (or the
equivalent C API) is the way to go.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm
place to shift.

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi

On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote:

> Dear Heiko Schocher,
> 
> In message <51A30F34.7030603@denx.de> you wrote:
>> 
>> But how to handle a raw nand partition and a ubi partition on one
>> nand?
>> 
>> If ubi is a new dfu interface, somebody must start dfu on u-boot
>> with "dfu nand .." or "dfu ubi .." dependent on which partition
>> has to be updated ... before using dfu-util on the host side ...
>> and start dfu-util for the correct partition...
>> 
>> This seems not really userfriendly to me ... if I have to use the
> 
> Indeed, this makes no sense and breaks the whole concept of DFU to be
> able to download a sinlge firmware image with one, simple command.
> 
>> Is ubi really a "interface" as nand or mmc ... ?
> 
> No, it is not.  It could be considered a "partition type" at best.
> 
>>> Looks like it's simple enough; erase (but don't step over the wear counters)
>>> , write (but skip over the wear counters).
>> 
>> Yep, or load the complete image in ram, and write it with "ubi write ..."
> 
> Not "or".  When dealing with UBI volumes, then "ubi write" (or the
> equivalent C API) is the way to go.
> 
> 

I pretty much agree. UBI looks like it's partition type.

BTW, the whole point of DFU is not to store your image in RAM at all.
There are very few systems that have that much RAM.

> Best regards,
> 
> Wolfgang Denk
> 
> -- 
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
> All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm
> place to shift.

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Mon, May 27, 2013 at 07:29:02PM +0300, Pantelis Antoniou wrote:
> Hi
> 
> On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote:
> 
> > Dear Heiko Schocher,
> > 
> > In message <51A30F34.7030603@denx.de> you wrote:
> >> 
> >> But how to handle a raw nand partition and a ubi partition on one
> >> nand?
> >> 
> >> If ubi is a new dfu interface, somebody must start dfu on u-boot
> >> with "dfu nand .." or "dfu ubi .." dependent on which partition
> >> has to be updated ... before using dfu-util on the host side ...
> >> and start dfu-util for the correct partition...
> >> 
> >> This seems not really userfriendly to me ... if I have to use the
> > 
> > Indeed, this makes no sense and breaks the whole concept of DFU to be
> > able to download a sinlge firmware image with one, simple command.
> > 
> >> Is ubi really a "interface" as nand or mmc ... ?
> > 
> > No, it is not.  It could be considered a "partition type" at best.
> > 
> >>> Looks like it's simple enough; erase (but don't step over the wear counters)
> >>> , write (but skip over the wear counters).
> >> 
> >> Yep, or load the complete image in ram, and write it with "ubi write ..."
> > 
> > Not "or".  When dealing with UBI volumes, then "ubi write" (or the
> > equivalent C API) is the way to go.
> > 
> > 
> 
> I pretty much agree. UBI looks like it's partition type.
> 
> BTW, the whole point of DFU is not to store your image in RAM at all.
> There are very few systems that have that much RAM.

This _may_ be the hard part for UBI.  When doing raw block writes for
NAND/MMC, we're able to write them out quickly and thus support images
larger than RAM.  But for filesystems we don't support that notion in
general for write and so limit ourselves to 8MiB or so files.  Fine for
the most part, but not fine for UBI.  It's possible that we can support
this on UBI easier than we can on filesystems, but I just don't know.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/2f384bf4/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Tom,

In message <20130527204127.GY17119@bill-the-cat> you wrote:
> 
> This _may_ be the hard part for UBI.  When doing raw block writes for
> NAND/MMC, we're able to write them out quickly and thus support images
> larger than RAM.  But for filesystems we don't support that notion in
> general for write and so limit ourselves to 8MiB or so files.  Fine for

Where exactly is this 8 MB limit coming into play?

> the most part, but not fine for UBI.  It's possible that we can support
> this on UBI easier than we can on filesystems, but I just don't know.

I thought the only size limitation for images we can load is available
system RAM?  Is this not the case?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
PoB = "Prisoner of Bill" -- those held captive, unwillingly or other-
wise, by the contemptible Microsoft monopoly.
         -- Tom Christiansen in <6abo45$3lc$2@csnews.cs.colorado.edu>

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Mon, May 27, 2013 at 11:25:52PM +0200, Wolfgang Denk wrote:

> Dear Tom,
> 
> In message <20130527204127.GY17119@bill-the-cat> you wrote:
> > 
> > This _may_ be the hard part for UBI.  When doing raw block writes for
> > NAND/MMC, we're able to write them out quickly and thus support images
> > larger than RAM.  But for filesystems we don't support that notion in
> > general for write and so limit ourselves to 8MiB or so files.  Fine for
> 
> Where exactly is this 8 MB limit coming into play?

In buffering the data.  We cannot write a chunk of a file to a
filesystem and then append to it, we don't have the API today.

> > the most part, but not fine for UBI.  It's possible that we can support
> > this on UBI easier than we can on filesystems, but I just don't know.
> 
> I thought the only size limitation for images we can load is available
> system RAM?  Is this not the case?

It is the case for raw media and not the case for filesystems.  If it
will be the case for UBI depends on what API we have available to us in
the UBI layer.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/ccb6cb98/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Tom,

In message <20130527233735.GZ17119@bill-the-cat> you wrote:
> 
> > Where exactly is this 8 MB limit coming into play?
> 
> In buffering the data.  We cannot write a chunk of a file to a
> filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in
thepast, so I really don't see where any buffer problem could be.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
No question is too silly to ask. Of course, some  questions  are  too
silly to to answer...  - L. Wall & R. L. Schwartz, _Programming Perl_

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
> > 
> > > Where exactly is this 8 MB limit coming into play?
> > 
> > In buffering the data.  We cannot write a chunk of a file to a
> > filesystem and then append to it, we don't have the API today.
> 
> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
> not load a 256 MiB file to RAM, and then write it to a file system?
> 
> I have definitely dealt with images and files bigger than 8 MiB in
> thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from,
after I sent the email.  The problem we have, and this is only for
writing to a filesystem (_not_ writing of a filesystem) is that we do
not have the API for appending to files, only create/overwrite.  So we
must read the whole file into memory, and then write it out.  The DFU
protocol doesn't have (I would swear anyhow) a part where it says "I'm
about to send you a blob of X bytes", so we cannot know at the start how
much data is coming our way.

Today we "solve" this with a statically defined
CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
be that same value.  Going forward, we may be able to switch this to
(and both of these are off the top of my head) a getenv to see how much
space to malloc, or just making it a malloc and adding some compile-time
check to ensure that the malloc area is at least as big as
CONFIG_SYS_DFU_MAX_FILE_SIZE.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/4a2e0ec7/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi Tom,

On May 28, 2013, at 6:01 PM, Tom Rini wrote:

> On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
>> Dear Tom,
>> 
>> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
>>> 
>>>> Where exactly is this 8 MB limit coming into play?
>>> 
>>> In buffering the data.  We cannot write a chunk of a file to a
>>> filesystem and then append to it, we don't have the API today.
>> 
>> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
>> not load a 256 MiB file to RAM, and then write it to a file system?
>> 
>> I have definitely dealt with images and files bigger than 8 MiB in
>> thepast, so I really don't see where any buffer problem could be.
> 
> I thought I might not have been clear about where this limit comes from,
> after I sent the email.  The problem we have, and this is only for
> writing to a filesystem (_not_ writing of a filesystem) is that we do
> not have the API for appending to files, only create/overwrite.  So we
> must read the whole file into memory, and then write it out.  The DFU
> protocol doesn't have (I would swear anyhow) a part where it says "I'm
> about to send you a blob of X bytes", so we cannot know at the start how
> much data is coming our way.
> 
> Today we "solve" this with a statically defined
> CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
> buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
> be that same value.  Going forward, we may be able to switch this to
> (and both of these are off the top of my head) a getenv to see how much
> space to malloc, or just making it a malloc and adding some compile-time
> check to ensure that the malloc area is at least as big as
> CONFIG_SYS_DFU_MAX_FILE_SIZE.
> 

Correct, the DFU protocol doesn't have a method to inform you before hand
about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment
variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our
thing.

> -- 
> Tom

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Benoît Thébaudeau]

Dear Pantelis Antoniou,

On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote:
> Hi Tom,
> 
> On May 28, 2013, at 6:01 PM, Tom Rini wrote:
> 
> > On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
> >> Dear Tom,
> >> 
> >> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
> >>> 
> >>>> Where exactly is this 8 MB limit coming into play?
> >>> 
> >>> In buffering the data.  We cannot write a chunk of a file to a
> >>> filesystem and then append to it, we don't have the API today.
> >> 
> >> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
> >> not load a 256 MiB file to RAM, and then write it to a file system?
> >> 
> >> I have definitely dealt with images and files bigger than 8 MiB in
> >> thepast, so I really don't see where any buffer problem could be.
> > 
> > I thought I might not have been clear about where this limit comes from,
> > after I sent the email.  The problem we have, and this is only for
> > writing to a filesystem (_not_ writing of a filesystem) is that we do
> > not have the API for appending to files, only create/overwrite.  So we
> > must read the whole file into memory, and then write it out.  The DFU
> > protocol doesn't have (I would swear anyhow) a part where it says "I'm
> > about to send you a blob of X bytes", so we cannot know at the start how
> > much data is coming our way.
> > 
> > Today we "solve" this with a statically defined
> > CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
> > buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
> > be that same value.  Going forward, we may be able to switch this to
> > (and both of these are off the top of my head) a getenv to see how much
> > space to malloc, or just making it a malloc and adding some compile-time
> > check to ensure that the malloc area is at least as big as
> > CONFIG_SYS_DFU_MAX_FILE_SIZE.
> > 
> 
> Correct, the DFU protocol doesn't have a method to inform you before hand
> about the size of the transfer about to happen.
> 
> The only possible solution I see at this point is to have an environment
> variable, i.e. dfubuf that controls the size of the buffer.
> 
> Upon start of a dfu transfer we can allocate the buffer, and do our
> thing.

I don't know the details of the DFU implementation in U-Boot, but the
specification leaves the choice between programming the firmware on-the-fly
during the download, and later during the manifestation phase (or a mix of
both). Hence, there is not need for a global firmware buffer if U-Boot goes for
the on-the-fly programming strategy. The only buffer constraint would be
wTransferSize (chosen by U-Boot for the control endpoint) in that case. See
"7. Manifestation Phase" on page 26 here:
http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf

Of course this can't yet apply to writing files on file systems since the
current API in U-Boot misses the append feature, but this could be applied to
program raw memory partitions, including UBI images.

Best regards,
Beno?t

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi Beno?t

On May 28, 2013, at 7:31 PM, Beno?t Th?baudeau wrote:

> Dear Pantelis Antoniou,
> 
> On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote:
>> Hi Tom,
>> 
>> On May 28, 2013, at 6:01 PM, Tom Rini wrote:
>> 
>>> On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
>>>> Dear Tom,
>>>> 
>>>> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
>>>>> 
>>>>>> Where exactly is this 8 MB limit coming into play?
>>>>> 
>>>>> In buffering the data.  We cannot write a chunk of a file to a
>>>>> filesystem and then append to it, we don't have the API today.
>>>> 
>>>> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
>>>> not load a 256 MiB file to RAM, and then write it to a file system?
>>>> 
>>>> I have definitely dealt with images and files bigger than 8 MiB in
>>>> thepast, so I really don't see where any buffer problem could be.
>>> 
>>> I thought I might not have been clear about where this limit comes from,
>>> after I sent the email.  The problem we have, and this is only for
>>> writing to a filesystem (_not_ writing of a filesystem) is that we do
>>> not have the API for appending to files, only create/overwrite.  So we
>>> must read the whole file into memory, and then write it out.  The DFU
>>> protocol doesn't have (I would swear anyhow) a part where it says "I'm
>>> about to send you a blob of X bytes", so we cannot know at the start how
>>> much data is coming our way.
>>> 
>>> Today we "solve" this with a statically defined
>>> CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
>>> buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
>>> be that same value.  Going forward, we may be able to switch this to
>>> (and both of these are off the top of my head) a getenv to see how much
>>> space to malloc, or just making it a malloc and adding some compile-time
>>> check to ensure that the malloc area is at least as big as
>>> CONFIG_SYS_DFU_MAX_FILE_SIZE.
>>> 
>> 
>> Correct, the DFU protocol doesn't have a method to inform you before hand
>> about the size of the transfer about to happen.
>> 
>> The only possible solution I see at this point is to have an environment
>> variable, i.e. dfubuf that controls the size of the buffer.
>> 
>> Upon start of a dfu transfer we can allocate the buffer, and do our
>> thing.
> 
> I don't know the details of the DFU implementation in U-Boot, but the
> specification leaves the choice between programming the firmware on-the-fly
> during the download, and later during the manifestation phase (or a mix of
> both). Hence, there is not need for a global firmware buffer if U-Boot goes for
> the on-the-fly programming strategy. The only buffer constraint would be
> wTransferSize (chosen by U-Boot for the control endpoint) in that case. See
> "7. Manifestation Phase" on page 26 here:
> http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf
> 

The problem is not DFU TBH, it's that since we don't have an option to append
to a file, we have to have the whole file transferred in RAM and written in
one go. The raw medium dfu methods in u-boot don't have a problem.

> Of course this can't yet apply to writing files on file systems since the
> current API in U-Boot misses the append feature, but this could be applied to
> program raw memory partitions, including UBI images.
> 

It already happens for raw memory partitions, it's the UBI images being discussed.
> Best regards,
> Beno?t

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Benoît Thébaudeau]

Hi Pantelis,

On Tuesday, May 28, 2013 6:43:06 PM, Pantelis Antoniou wrote:
> Hi Beno?t
> 
> On May 28, 2013, at 7:31 PM, Beno?t Th?baudeau wrote:
> 
> > Dear Pantelis Antoniou,
> > 
> > On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote:
> >> Hi Tom,
> >> 
> >> On May 28, 2013, at 6:01 PM, Tom Rini wrote:
> >> 
> >>> On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
> >>>> Dear Tom,
> >>>> 
> >>>> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
> >>>>> 
> >>>>>> Where exactly is this 8 MB limit coming into play?
> >>>>> 
> >>>>> In buffering the data.  We cannot write a chunk of a file to a
> >>>>> filesystem and then append to it, we don't have the API today.
> >>>> 
> >>>> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
> >>>> not load a 256 MiB file to RAM, and then write it to a file system?
> >>>> 
> >>>> I have definitely dealt with images and files bigger than 8 MiB in
> >>>> thepast, so I really don't see where any buffer problem could be.
> >>> 
> >>> I thought I might not have been clear about where this limit comes from,
> >>> after I sent the email.  The problem we have, and this is only for
> >>> writing to a filesystem (_not_ writing of a filesystem) is that we do
> >>> not have the API for appending to files, only create/overwrite.  So we
> >>> must read the whole file into memory, and then write it out.  The DFU
> >>> protocol doesn't have (I would swear anyhow) a part where it says "I'm
> >>> about to send you a blob of X bytes", so we cannot know at the start how
> >>> much data is coming our way.
> >>> 
> >>> Today we "solve" this with a statically defined
> >>> CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
> >>> buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
> >>> be that same value.  Going forward, we may be able to switch this to
> >>> (and both of these are off the top of my head) a getenv to see how much
> >>> space to malloc, or just making it a malloc and adding some compile-time
> >>> check to ensure that the malloc area is at least as big as
> >>> CONFIG_SYS_DFU_MAX_FILE_SIZE.
> >>> 
> >> 
> >> Correct, the DFU protocol doesn't have a method to inform you before hand
> >> about the size of the transfer about to happen.
> >> 
> >> The only possible solution I see at this point is to have an environment
> >> variable, i.e. dfubuf that controls the size of the buffer.
> >> 
> >> Upon start of a dfu transfer we can allocate the buffer, and do our
> >> thing.
> > 
> > I don't know the details of the DFU implementation in U-Boot, but the
> > specification leaves the choice between programming the firmware on-the-fly
> > during the download, and later during the manifestation phase (or a mix of
> > both). Hence, there is not need for a global firmware buffer if U-Boot goes
> > for
> > the on-the-fly programming strategy. The only buffer constraint would be
> > wTransferSize (chosen by U-Boot for the control endpoint) in that case. See
> > "7. Manifestation Phase" on page 26 here:
> > http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf
> > 
> 
> The problem is not DFU TBH, it's that since we don't have an option to append
> to a file, we have to have the whole file transferred in RAM and written in
> one go. The raw medium dfu methods in u-boot don't have a problem.
> 
> > Of course this can't yet apply to writing files on file systems since the
> > current API in U-Boot misses the append feature, but this could be applied
> > to
> > program raw memory partitions, including UBI images.
> > 
> 
> It already happens for raw memory partitions, it's the UBI images being
> discussed.

But what does appending to a file has to do with programming a UBI image, which
is a memory partition containing a whole file system? This is what I don't get
in this discussion. Is it because of a restriction of the DFU API in U-Boot?

Best regards,
Beno?t

[U-Boot] dfu: dfu and UBI Volumes

[Pantelis Antoniou]

Hi 
On May 28, 2013, at 7:43 PM, Beno?t Th?baudeau wrote:

> Hi Pantelis,
> 
> On Tuesday, May 28, 2013 6:43:06 PM, Pantelis Antoniou wrote:
>> Hi Beno?t
>> 
>> On May 28, 2013, at 7:31 PM, Beno?t Th?baudeau wrote:
>> 
>>> Dear Pantelis Antoniou,
>>> 
>>> On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote:
>>>> Hi Tom,
>>>> 
>>>> On May 28, 2013, at 6:01 PM, Tom Rini wrote:
>>>> 
>>>>> On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote:
>>>>>> Dear Tom,
>>>>>> 
>>>>>> In message <20130527233735.GZ17119@bill-the-cat> you wrote:
>>>>>>> 
>>>>>>>> Where exactly is this 8 MB limit coming into play?
>>>>>>> 
>>>>>>> In buffering the data.  We cannot write a chunk of a file to a
>>>>>>> filesystem and then append to it, we don't have the API today.
>>>>>> 
>>>>>> Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I
>>>>>> not load a 256 MiB file to RAM, and then write it to a file system?
>>>>>> 
>>>>>> I have definitely dealt with images and files bigger than 8 MiB in
>>>>>> thepast, so I really don't see where any buffer problem could be.
>>>>> 
>>>>> I thought I might not have been clear about where this limit comes from,
>>>>> after I sent the email.  The problem we have, and this is only for
>>>>> writing to a filesystem (_not_ writing of a filesystem) is that we do
>>>>> not have the API for appending to files, only create/overwrite.  So we
>>>>> must read the whole file into memory, and then write it out.  The DFU
>>>>> protocol doesn't have (I would swear anyhow) a part where it says "I'm
>>>>> about to send you a blob of X bytes", so we cannot know at the start how
>>>>> much data is coming our way.
>>>>> 
>>>>> Today we "solve" this with a statically defined
>>>>> CONFIG_SYS_DFU_MAX_FILE_SIZE.  Looking at things again, I think this is
>>>>> buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also
>>>>> be that same value.  Going forward, we may be able to switch this to
>>>>> (and both of these are off the top of my head) a getenv to see how much
>>>>> space to malloc, or just making it a malloc and adding some compile-time
>>>>> check to ensure that the malloc area is at least as big as
>>>>> CONFIG_SYS_DFU_MAX_FILE_SIZE.
>>>>> 
>>>> 
>>>> Correct, the DFU protocol doesn't have a method to inform you before hand
>>>> about the size of the transfer about to happen.
>>>> 
>>>> The only possible solution I see at this point is to have an environment
>>>> variable, i.e. dfubuf that controls the size of the buffer.
>>>> 
>>>> Upon start of a dfu transfer we can allocate the buffer, and do our
>>>> thing.
>>> 
>>> I don't know the details of the DFU implementation in U-Boot, but the
>>> specification leaves the choice between programming the firmware on-the-fly
>>> during the download, and later during the manifestation phase (or a mix of
>>> both). Hence, there is not need for a global firmware buffer if U-Boot goes
>>> for
>>> the on-the-fly programming strategy. The only buffer constraint would be
>>> wTransferSize (chosen by U-Boot for the control endpoint) in that case. See
>>> "7. Manifestation Phase" on page 26 here:
>>> http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf
>>> 
>> 
>> The problem is not DFU TBH, it's that since we don't have an option to append
>> to a file, we have to have the whole file transferred in RAM and written in
>> one go. The raw medium dfu methods in u-boot don't have a problem.
>> 
>>> Of course this can't yet apply to writing files on file systems since the
>>> current API in U-Boot misses the append feature, but this could be applied
>>> to
>>> program raw memory partitions, including UBI images.
>>> 
>> 
>> It already happens for raw memory partitions, it's the UBI images being
>> discussed.
> 
> But what does appending to a file has to do with programming a UBI image, which
> is a memory partition containing a whole file system? This is what I don't get
> in this discussion. Is it because of a restriction of the DFU API in U-Boot?
> 

Don't expect a discussion on a mailing list to stay on topic for long :)
We sort of drifted from UBI to the fixed sized buffer.

> Best regards,
> Beno?t

Regards

-- Pantelis

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Tue, May 28, 2013 at 06:31:41PM +0200, Beno??t Th??baudeau wrote:

[snip]
> Of course this can't yet apply to writing files on file systems since the
> current API in U-Boot misses the append feature, but this could be applied to
> program raw memory partitions, including UBI images.

Correct.  We can write a whole UBI image, today, of NAND size,
regardless of DDR size.  But modifying UBI volumes (so UBIFS or your
kernel in UBI or ...) will depend on what the UBI API provides us today.
Modifying files on UBIFS (say replacing the kernel that's in UBIFS
rather than a UBI volume itself) is another wrinkle, due to a lack of
filesystem append.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/849ae00d/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Tom,

In message <20130528172309.GF5829@bill-the-cat> you wrote:
> 
> > Of course this can't yet apply to writing files on file systems since the
> > current API in U-Boot misses the append feature, but this could be applied to
> > program raw memory partitions, including UBI images.
>
> Correct.  We can write a whole UBI image, today, of NAND size,
> regardless of DDR size.  But modifying UBI volumes (so UBIFS or your

I don't think so.  To write a UBI volume on an existing UBI device,
you would use the "ubi write" command.  This translates into a call of
ubi_volume_write(char *volume, void *buf, size_t size)  which means
the size must be known before you start writing; as far as I can tell
ubi_volume_write() does not support incremental write operations of
individual "parts" of a volume.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Its always easier short term to pee in the pond
than install a toilet - it's just not a good long term plan.
          - Alan Cox in <20100101145701.6432e7b7@lxorguk.ukuu.org.uk>

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote:
> Dear Tom,
> 
> In message <20130528172309.GF5829@bill-the-cat> you wrote:
> > 
> > > Of course this can't yet apply to writing files on file systems since the
> > > current API in U-Boot misses the append feature, but this could be applied to
> > > program raw memory partitions, including UBI images.
> >
> > Correct.  We can write a whole UBI image, today, of NAND size,
> > regardless of DDR size.  But modifying UBI volumes (so UBIFS or your
> 
> I don't think so.  To write a UBI volume on an existing UBI device,
> you would use the "ubi write" command.  This translates into a call of
> ubi_volume_write(char *volume, void *buf, size_t size)  which means
> the size must be known before you start writing; as far as I can tell
> ubi_volume_write() does not support incremental write operations of
> individual "parts" of a volume.

OK.  A very quick read of ubi_volume_write leads into the guts of the
write being in drivers/mtd/ubi/upd.c and it feels like you could
actually do the volume write in chunks with ubi_start_update() and
ubi_more_update_data() (and maybe some other housekeeping bits).  It
might take a bit more work since it looks like looks like both functions
rely on knowing the size at the start, but that's just to make sure the
size will fit in the volume.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/7edf2c92/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Tom,

Am 28.05.2013 23:16, schrieb Tom Rini:
> On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote:
>> Dear Tom,
>>
>> In message <20130528172309.GF5829@bill-the-cat> you wrote:
>>>
>>>> Of course this can't yet apply to writing files on file systems since the
>>>> current API in U-Boot misses the append feature, but this could be applied to
>>>> program raw memory partitions, including UBI images.
>>>
>>> Correct.  We can write a whole UBI image, today, of NAND size,
>>> regardless of DDR size.  But modifying UBI volumes (so UBIFS or your
>>
>> I don't think so.  To write a UBI volume on an existing UBI device,
>> you would use the "ubi write" command.  This translates into a call of
>> ubi_volume_write(char *volume, void *buf, size_t size)  which means
>> the size must be known before you start writing; as far as I can tell
>> ubi_volume_write() does not support incremental write operations of
>> individual "parts" of a volume.
> 
> OK.  A very quick read of ubi_volume_write leads into the guts of the
> write being in drivers/mtd/ubi/upd.c and it feels like you could
> actually do the volume write in chunks with ubi_start_update() and
> ubi_more_update_data() (and maybe some other housekeeping bits).  It
> might take a bit more work since it looks like looks like both functions
> rely on knowing the size at the start, but that's just to make sure the
> size will fit in the volume.

Yes, I think so too ... seems some work, but not unsolveable ...

Hmm.. is it possible to get the filesize over dfu on startup? (I hope
so, as it makes no sense to transfer a file, if it does not fit in the
partition ... )

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Wed, May 29, 2013 at 06:35:27AM +0200, Heiko Schocher wrote:

> Hello Tom,
> 
> Am 28.05.2013 23:16, schrieb Tom Rini:
> > On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote:
> >> Dear Tom,
> >>
> >> In message <20130528172309.GF5829@bill-the-cat> you wrote:
> >>>
> >>>> Of course this can't yet apply to writing files on file systems since the
> >>>> current API in U-Boot misses the append feature, but this could be applied to
> >>>> program raw memory partitions, including UBI images.
> >>>
> >>> Correct.  We can write a whole UBI image, today, of NAND size,
> >>> regardless of DDR size.  But modifying UBI volumes (so UBIFS or your
> >>
> >> I don't think so.  To write a UBI volume on an existing UBI device,
> >> you would use the "ubi write" command.  This translates into a call of
> >> ubi_volume_write(char *volume, void *buf, size_t size)  which means
> >> the size must be known before you start writing; as far as I can tell
> >> ubi_volume_write() does not support incremental write operations of
> >> individual "parts" of a volume.
> > 
> > OK.  A very quick read of ubi_volume_write leads into the guts of the
> > write being in drivers/mtd/ubi/upd.c and it feels like you could
> > actually do the volume write in chunks with ubi_start_update() and
> > ubi_more_update_data() (and maybe some other housekeeping bits).  It
> > might take a bit more work since it looks like looks like both functions
> > rely on knowing the size at the start, but that's just to make sure the
> > size will fit in the volume.
> 
> Yes, I think so too ... seems some work, but not unsolveable ...
> 
> Hmm.. is it possible to get the filesize over dfu on startup? (I hope
> so, as it makes no sense to transfer a file, if it does not fit in the
> partition ... )

There is not, but that's OK.  Even if you think you had the room for the
whole image you could find a new bad block and not fit, so it's just
behaving like when we do a raw NAND write, get the data chunk, make sure
we think we still have room, write if we do, error if we don't.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130529/7a06f8da/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Wolfgang,

Am 27.05.2013 23:25, schrieb Wolfgang Denk:
> Dear Tom,
> 
> In message <20130527204127.GY17119@bill-the-cat> you wrote:
>>
>> This _may_ be the hard part for UBI.  When doing raw block writes for
>> NAND/MMC, we're able to write them out quickly and thus support images
>> larger than RAM.  But for filesystems we don't support that notion in
>> general for write and so limit ourselves to 8MiB or so files.  Fine for
> 
> Where exactly is this 8 MB limit coming into play?

You find this in drivers/dfu/dfu.c:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
                                   dfu_buf[DFU_DATA_BUF_SIZE];

I had just a patch in my workqueue, where I made this value
at least configurable ... but this limits not the transfer file
size, see below...

>> the most part, but not fine for UBI.  It's possible that we can support
>> this on UBI easier than we can on filesystems, but I just don't know.
> 
> I thought the only size limitation for images we can load is available
> system RAM?  Is this not the case?

As I read in code: Raw nand partitions fill the above buffer, if
it is full -> flush to nand, and go on ... so no limit for them.

drivers/dfu/dfu_mmc.c use (another?, why?) buffer:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
                                dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];

and use this buffer for not raw partitions ... and this buffer
gets flushed, only if the complete file is transfered, as the
README states:

                CONFIG_SYS_DFU_MAX_FILE_SIZE
                When updating files rather than the raw storage device,
                we use a static buffer to copy the file into and then write
                the buffer once we've been given the whole file.  Define
                this to the maximum filesize (in bytes) for the buffer.
                Default is 4 MiB if undefined.

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Heiko,

In message <51A427A8.8090709@denx.de> you wrote:
>
> > Where exactly is this 8 MB limit coming into play?
> 
> You find this in drivers/dfu/dfu.c:
> 
> static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
>                                    dfu_buf[DFU_DATA_BUF_SIZE];

Ah, so it is a DFU restriction!

Well, this is a design problem that needs to be fixed.

> As I read in code: Raw nand partitions fill the above buffer, if
> it is full -> flush to nand, and go on ... so no limit for them.

So this boils down to the question whether there is some "incremental
ubi write" method.

Hello UBI experts?

> drivers/dfu/dfu_mmc.c use (another?, why?) buffer:
> 
> static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
>                                 dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];
...
> and use this buffer for not raw partitions ... and this buffer
> gets flushed, only if the complete file is transfered, as the
> README states:
> 
>                 CONFIG_SYS_DFU_MAX_FILE_SIZE
>                 When updating files rather than the raw storage device,
>                 we use a static buffer to copy the file into and then write
>                 the buffer once we've been given the whole file.  Define
>                 this to the maximum filesize (in bytes) for the buffer.
>                 Default is 4 MiB if undefined.

This makes very little sense to me.  Why do we need another (and even
smaller) buffer when we already have one?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"We don't have to protect the environment -- the Second Coming is  at
hand."                                                   - James Watt

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Tue, May 28, 2013 at 07:55:03AM +0200, Wolfgang Denk wrote:
> Dear Heiko,
> 
> In message <51A427A8.8090709@denx.de> you wrote:
> >
> > > Where exactly is this 8 MB limit coming into play?
> > 
> > You find this in drivers/dfu/dfu.c:
> > 
> > static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
> >                                    dfu_buf[DFU_DATA_BUF_SIZE];
> 
> Ah, so it is a DFU restriction!
[snip]
> > drivers/dfu/dfu_mmc.c use (another?, why?) buffer:
> > 
> > static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)
> >                                 dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];
> ...
> > and use this buffer for not raw partitions ... and this buffer
> > gets flushed, only if the complete file is transfered, as the
> > README states:
> > 
> >                 CONFIG_SYS_DFU_MAX_FILE_SIZE
> >                 When updating files rather than the raw storage device,
> >                 we use a static buffer to copy the file into and then write
> >                 the buffer once we've been given the whole file.  Define
> >                 this to the maximum filesize (in bytes) for the buffer.
> >                 Default is 4 MiB if undefined.
> 
> This makes very little sense to me.  Why do we need another (and even
> smaller) buffer when we already have one?

Per my other email, the intention and implementation didn't quite
match-up.  The intent of the README should be (but isn't) reflected) in
the code.  And perhaps we can come up with something better than a big
static allocation.  Perhaps.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/9ead52ad/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Tom,

Am 27.05.2013 22:41, schrieb Tom Rini:
> On Mon, May 27, 2013 at 07:29:02PM +0300, Pantelis Antoniou wrote:
>> Hi
>>
>> On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote:
>>
>>> Dear Heiko Schocher,
>>>
>>> In message <51A30F34.7030603@denx.de> you wrote:
[...]
>> I pretty much agree. UBI looks like it's partition type.
>>
>> BTW, the whole point of DFU is not to store your image in RAM at all.
>> There are very few systems that have that much RAM.
> 
> This _may_ be the hard part for UBI.  When doing raw block writes for
> NAND/MMC, we're able to write them out quickly and thus support images
> larger than RAM.  But for filesystems we don't support that notion in
> general for write and so limit ourselves to 8MiB or so files.  Fine for
> the most part, but not fine for UBI.  It's possible that we can support
> this on UBI easier than we can on filesystems, but I just don't know.

I also do not know this at the moment :-(

Maybe a UBI expert can help here?

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Sun, May 26, 2013 at 09:09:22AM +0200, Heiko Schocher wrote:

[snip]
> Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout
> for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...
> 
> After all ... should we add a DFU_UBI and add this to
> drivers/dfu/dfu_nand.c?

Yes, I think we should be able to adapt dfu_nand to support raw
(current) and UBI (which will need a little further handling so you can
update per UBI container).  In MMC (and there's examples in trats and
am335x_evm) we say <name> ext4/fat/part/raw device#/start_blk part#/end_blk.
I would imagine, but testing and implementation might show a better way,
we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and
fat/ext knowledge.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/32a73a89/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Tom,

Am 27.05.2013 21:21, schrieb Tom Rini:
> On Sun, May 26, 2013 at 09:09:22AM +0200, Heiko Schocher wrote:
> 
> [snip]
>> Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout
>> for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...
>>
>> After all ... should we add a DFU_UBI and add this to
>> drivers/dfu/dfu_nand.c?
> 
> Yes, I think we should be able to adapt dfu_nand to support raw
> (current) and UBI (which will need a little further handling so you can
> update per UBI container).  In MMC (and there's examples in trats and
> am335x_evm) we say <name> ext4/fat/part/raw device#/start_blk part#/end_blk.
> I would imagine, but testing and implementation might show a better way,
> we do UBI as <name> ubi ubiN volume-name, ie:
> rootfs ubi ubi0 rootfs
> user ubi ubi0 user-data
> and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and
> fat/ext knowledge.

Yes, I think, thats the way to go ...

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Heiko,

In message <51A42CCD.1020607@denx.de> you wrote:
>
> > I would imagine, but testing and implementation might show a better way,
> > we do UBI as <name> ubi ubiN volume-name, ie:
> > rootfs ubi ubi0 rootfs
> > user ubi ubi0 user-data
> > and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and
> > fat/ext knowledge.
> 
> Yes, I think, thats the way to go ...

I doubt that dfu_nand.c is the right place for this.  What if I start
using DFU on NOR flash?  The decisions which device type (NAND, MMC,
NOR, USB mass storage, ...) to habdle on one side, and which partition
type / image type (raw, UBI volume, file, ...) on the other side are
fully orthogonal to each other.  They should be handled by independent
code, and not one of them repeated for all implementations of the
other.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
A mouse is an elephant built by the Japanese.

[U-Boot] dfu: dfu and UBI Volumes

[Heiko Schocher]

Hello Wolfgang,

Am 28.05.2013 07:58, schrieb Wolfgang Denk:
> Dear Heiko,
> 
> In message <51A42CCD.1020607@denx.de> you wrote:
>>
>>> I would imagine, but testing and implementation might show a better way,
>>> we do UBI as <name> ubi ubiN volume-name, ie:
>>> rootfs ubi ubi0 rootfs
>>> user ubi ubi0 user-data
>>> and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and
>>> fat/ext knowledge.
>>
>> Yes, I think, thats the way to go ...
> 
> I doubt that dfu_nand.c is the right place for this.  What if I start
> using DFU on NOR flash?  The decisions which device type (NAND, MMC,
> NOR, USB mass storage, ...) to habdle on one side, and which partition
> type / image type (raw, UBI volume, file, ...) on the other side are
> fully orthogonal to each other.  They should be handled by independent
> code, and not one of them repeated for all implementations of the
> other.

Yes, exactly ...

bye,
Heiko
-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

[U-Boot] dfu: dfu and UBI Volumes

[Tom Rini]

On Tue, May 28, 2013 at 08:24:06AM +0200, Heiko Schocher wrote:
> Hello Wolfgang,
> 
> Am 28.05.2013 07:58, schrieb Wolfgang Denk:
> > Dear Heiko,
> > 
> > In message <51A42CCD.1020607@denx.de> you wrote:
> >>
> >>> I would imagine, but testing and implementation might show a better way,
> >>> we do UBI as <name> ubi ubiN volume-name, ie:
> >>> rootfs ubi ubi0 rootfs
> >>> user ubi ubi0 user-data
> >>> and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and
> >>> fat/ext knowledge.
> >>
> >> Yes, I think, thats the way to go ...
> > 
> > I doubt that dfu_nand.c is the right place for this.  What if I start
> > using DFU on NOR flash?  The decisions which device type (NAND, MMC,
> > NOR, USB mass storage, ...) to habdle on one side, and which partition
> > type / image type (raw, UBI volume, file, ...) on the other side are
> > fully orthogonal to each other.  They should be handled by independent
> > code, and not one of them repeated for all implementations of the
> > other.
> 
> Yes, exactly ...

We can see about what re-org makes sense once we've got another
implementation here.  We might be able to share the filesystem-based
write code, but that in part might cause some screams since it'll depend
on our continued (ab)use of run_command.  The device-specific code does
end up being the device-specific API part.   Unless we're adding ubifs
in addition to ubi volume support to DFU, we don't have real shared
filesystem stuff, yet.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/6494f8a1/attachment.pgp>

[U-Boot] dfu: dfu and UBI Volumes

[Wolfgang Denk]

Dear Pantelis Antoniou,

In message <6AD958CB-3CFC-4362-B72D-511147D041AC@antoniou-consulting.com> you wrote:
> 
> > How To do this? Current state on this board is to erase the rootfs
> > mtd partition with a nand erase and write the new image using
> > dfu_nand.c ... which calls in the end nand_write ... which is ...
> > lets say ... not the prefered way on an UBI volume ...
> > 
> > How to solve this? Any ideas?
> 
> Well, what would you like ideally to do? Why is nand_write not ideal for
> a UBI volume.

The key problem is that nand_erase will destroy all UBI wear levelling
and erase counters.  When working with UBI, you NEVER want to do this.

See for example [1] for reference.

> Note that dfu will skip over the bad blocks... 

Which is even worse here.

DFU should be able to deal with UBI volumes, and using proper access
routines (ubi write) to write the new volume(s).  But never ever
should any low level erase of the underlying flash device be needed
nor used.

[1] http://www.linux-mtd.infradead.org/doc/ubi.html#L_format

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
What's the sound a name makes when it's dropped?