Patching mkfs.ubifs.c for running under Windows ?

Patching mkfs.ubifs.c for running under Windows ?

[gutemine]

Hi!

I'm the author of ubidump, the first real ubifs image extract tool.
ubidump allows to extract ubifs images created by mkfs.ubifs and also ubifs
volumes created by ubinize. 

ubidump works without any nandsim, block2mtd or mtdram modules and doesn't
mount the image at all. Therefore it runs nicely on both Linux and Windows
as it is written in pure C and doesn't use any mtd-utils code or includes.

Live under Linux was easy, but I took the challange to make it run under
Windows too, because there it was not possible to mount and extract an
ubifs image until now.

ubidump under Windows supports Hard and Symlinks by using the native NTFS
functionality. Only device files and pipes are written as simple text files
containing the needed information for a re-pack (eg. char or block device
and major and minor number). 

As users of ubidump under Windows now asked me for getting also the
possibility to re-pack their extracted ubifs images, I'm considering to
patch mkfs.ubifs.c, so that it handles also the Windows hard and softlinks
and my dummy device files correctly. 

My question is now, if it would be OK to add these changes to the standard
mkfs.ubifs.c (off cource nicely wrapped in #ifdef WINDOWS statements so
that these changes don't do any harm when compiled on other plattforms) and
send you the diff, or if I should create an extra open source tool based on
the existing mkfs.ubifs.c code.

BTW I'm ware of the mkfs.ubifs cygwin implementation, but as explained
above I would prefere to make mkfs.ubifs run in native Windows as my
ubidump does.

Please advice on your preferences and in what direction I should proceede
with this project!

Kind regards,
gutemine

Re: Patching mkfs.ubifs.c for running under Windows ?

[richard -rw- weinberger]

On Sat, Sep 21, 2013 at 12:58 PM, gutemine <gutemine@oozoon.de> wrote:
> Hi!
>
> I'm the author of ubidump, the first real ubifs image extract tool.
> ubidump allows to extract ubifs images created by mkfs.ubifs and also ubifs
> volumes created by ubinize.

Where did you hide it?
Sounds interesting...

> ubidump works without any nandsim, block2mtd or mtdram modules and doesn't
> mount the image at all. Therefore it runs nicely on both Linux and Windows
> as it is written in pure C and doesn't use any mtd-utils code or includes.
>
> Live under Linux was easy, but I took the challange to make it run under
> Windows too, because there it was not possible to mount and extract an
> ubifs image until now.
>
> ubidump under Windows supports Hard and Symlinks by using the native NTFS
> functionality. Only device files and pipes are written as simple text files
> containing the needed information for a re-pack (eg. char or block device
> and major and minor number).
>
> As users of ubidump under Windows now asked me for getting also the
> possibility to re-pack their extracted ubifs images, I'm considering to
> patch mkfs.ubifs.c, so that it handles also the Windows hard and softlinks
> and my dummy device files correctly.
>
> My question is now, if it would be OK to add these changes to the standard
> mkfs.ubifs.c (off cource nicely wrapped in #ifdef WINDOWS statements so
> that these changes don't do any harm when compiled on other plattforms) and
> send you the diff, or if I should create an extra open source tool based on
> the existing mkfs.ubifs.c code.

-ENOPATCH. :-)

IOW, we need to review it first.

-- 
Thanks,
//richard

Re: Patching mkfs.ubifs.c for running under Windows ?

[gutemine]

Dear Richard!

See my comments below.

> Where did you hide it?

I didn't hide it, the ubidump development project just left testing
phase with a first release two weeks ago. I'm coming from the Dreambox
community and we develop plugins for our Linux based SAT receivers, 
therefore our path simply hasn't crossed yet.
 
> Sounds interesting...

Dreamboxes use OpenEmbedded Linux and recently switched their kernel to
3.2 and their Flash Filesystem from jffs2 to ubifs. For booting multiple
images we now needed to have the possibility to extract the ubifs images
directly to an ext4 filesystem on USB sticks.

These receivers often have only 256MB Memory and most of this is already
used. Therefore we soon ran out of memory when extracting images > 128MB
with the existing approaches to mount them via block2mtd or nandsim.
ubiattach of such simulated Flash simply terminated with out of memory
errors in dmesg. 

Therefore I was forced to solve the problem of direct UBIFS extraction
without any emulated Flashdevices and ubifs mounting. ubidump now does
this job nicely and needs less then 2MB of Memory during the extraction,
no matter how big the ubifs image is unless you use the caching option
to reduce IO by loading the whole ubifs image to ram and get faster
extraction on a PC. Then a typical 100 MB ubifs image on my Linux PC is
extracted in 2-3 seconds with about 6000 files. 

You can read the full story of the development project thread at
ubidump.oozoon.de and there you can download also testing kits for
Windows, Linux and mipsel as the Dreamboxes are using Broadcom CPUs. As
ubidump is written in pure C I could easily compile it also for other
plattforms on request.

But be aware due to the lack of help I received from my community I
decided to make ubidump shareware, but you are free to try it for 30
days.

> 
> -ENOPATCH. :-)
> 

I reviwed the changes done for the cygwin version and adding the /dev
and Windows hard and softlink handling would be only another 1-2
codepages, but I agree that it would NOT make live and code maintenance
easier.

> IOW, we need to review it first.
> 

Feel free to test ubidump and if you extract an image under Windows to
an NTFS directory check /dev files to see how I implemented the dummy
files so that you get an idea how mkfs.ubifs would need to read them
when re-creating the ubifs image. Unter Linux off course all files are
dumped natively.

PS: Your FAQ is wrong, now there exists an user space ubifs dump utility
on this planet.

Kind Regards
gutemine

Re: Patching mkfs.ubifs.c for running under Windows ?

[Richard Weinberger]

Hi *Would be nice to know your real name*!

Am 22.09.2013 11:19, schrieb gutemine:
> Dear Richard!
> 
> See my comments below.
> 
>> Where did you hide it?
> 
> I didn't hide it, the ubidump development project just left testing
> phase with a first release two weeks ago. I'm coming from the Dreambox
> community and we develop plugins for our Linux based SAT receivers, 
> therefore our path simply hasn't crossed yet.

Where can I find the sources?

> PS: Your FAQ is wrong, now there exists an user space ubifs dump utility
> on this planet.

Feel free to send a patch against
http://git.infradead.org/mtd-www.git

Thanks,
//richard

Re: Patching mkfs.ubifs.c for running under Windows ?

[gutemine]

Hi!

See my comments below.

> Hi *Would be nice to know your real name*!

The Dreambox community has areas of interrest which are
working on receiving PayTV with receivers not certifed by the
PayTV vendors, therefore people prefere to be known only by
their nicks. 

My focus areas are not any of these activities, but 
please respect my privacy. If needed I can drop you 
a mail from my personal account
but not via the full distrubution list.

> Where can I find the sources?

Originally it was planned to submit ubidump.c as a new member of mtd-utils
but as I received no real help from my community in testing and developing
addons to ubidump like a GUI for Windows and Linux, I decided to keep 
it closed source for the moment and ship it only as compiled binaries 
with a shareware license.

This is possible as I wrote ubidump from scratch after failing multiply
to create an ubidump tool out of existing code. 

It was simply a too complex task for me, so I had to do it my 
way - and I didn't even read much of the ubifs documentation either. 
To be honest this idiots approach reduced the complexity of the task
dramatically, 
but it still was a 200h project.

For adapting mkfs.ubifs to our needs I would off course respect the GPL
and publish the sources or a patch - which brings us back to my 
original question to the mailing list.
 
> Feel free to send a patch against
> http://git.infradead.org/mtd-www.git

I will try, but as I'm too old for using such modern stuff like git
this could take some time to find out how to diff the xml

Kind Regards,
gutemine

Re: Patching mkfs.ubifs.c for running under Windows ?

[Richard Weinberger]

Am 22.09.2013 12:00, schrieb gutemine:
>> Where can I find the sources?
> 
> Originally it was planned to submit ubidump.c as a new member of mtd-utils
> but as I received no real help from my community in testing and developing
> addons to ubidump like a GUI for Windows and Linux, I decided to keep 
> it closed source for the moment and ship it only as compiled binaries 
> with a shareware license.

Stopped reading here, sorry.
//richard

Re: Patching mkfs.ubifs.c for running under Windows ?

[gutemine]

Dear Richard!

> Stopped reading here, sorry.
> //richard

Thanks for accepting my decision without arguing.

Kind regards,
gutemine

Re: Patching mkfs.ubifs.c for running under Windows ?

[Artem Bityutskiy]

On Sat, 2013-09-21 at 12:58 +0200, gutemine wrote:
> As users of ubidump under Windows now asked me for getting also the
> possibility to re-pack their extracted ubifs images, I'm considering to
> patch mkfs.ubifs.c, so that it handles also the Windows hard and softlinks
> and my dummy device files correctly.

Generally, I am OK with making the code support windows, why not? But
this should be done very nicely and should not be intrusive. I guess you
should send patches and then we'll see.

-- 
Best Regards,
Artem Bityutskiy

Re: Patching mkfs.ubifs.c for running under Windows ?

[Florian Fainelli]

Hello,

2013/10/26 Artem Bityutskiy <dedekind1@gmail.com>:
> On Sat, 2013-09-21 at 12:58 +0200, gutemine wrote:
>> As users of ubidump under Windows now asked me for getting also the
>> possibility to re-pack their extracted ubifs images, I'm considering to
>> patch mkfs.ubifs.c, so that it handles also the Windows hard and softlinks
>> and my dummy device files correctly.
>
> Generally, I am OK with making the code support windows, why not? But
> this should be done very nicely and should not be intrusive. I guess you
> should send patches and then we'll see.

We do have some patches in OpenWrt to make mtd-utils build and work
under FreeBSD and Cygwin here:

https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches

in particular:
https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/110-portability.patch
https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/120-cygwin_fixes.patch
https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/134-freebsd_loff_t.patch

Not sure how acceptable they look like to you?
-- 
Florian

Re: Patching mkfs.ubifs.c for running under Windows ?

[Richard Weinberger]

On Mon, Oct 28, 2013 at 2:38 AM, Florian Fainelli <florian@openwrt.org> wrote:
> Hello,
>
> 2013/10/26 Artem Bityutskiy <dedekind1@gmail.com>:
>> On Sat, 2013-09-21 at 12:58 +0200, gutemine wrote:
>>> As users of ubidump under Windows now asked me for getting also the
>>> possibility to re-pack their extracted ubifs images, I'm considering to
>>> patch mkfs.ubifs.c, so that it handles also the Windows hard and softlinks
>>> and my dummy device files correctly.
>>
>> Generally, I am OK with making the code support windows, why not? But
>> this should be done very nicely and should not be intrusive. I guess you
>> should send patches and then we'll see.
>
> We do have some patches in OpenWrt to make mtd-utils build and work
> under FreeBSD and Cygwin here:
>
> https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches
>
> in particular:
> https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/110-portability.patch
> https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/120-cygwin_fixes.patch
> https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/134-freebsd_loff_t.patch
>
> Not sure how acceptable they look like to you?

Looks not that bad. :-)
But why do you need the file "include/cygwin/bits-byteswap.h"?

Artem, what do you think?
IMHO adding cygwin support is less intrusive than adding native win32 support.

-- 
Thanks,
//richard

Re: Patching mkfs.ubifs.c for running under Windows ?

[Artem Bityutskiy]

On Mon, 2013-10-28 at 11:47 +0100, Richard Weinberger wrote:
> > https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches
> >
> > in particular:
> > https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/110-portability.patch
> > https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/120-cygwin_fixes.patch
> > https://dev.openwrt.org/browser/trunk/tools/mtd-utils/patches/134-freebsd_loff_t.patch
> >
> > Not sure how acceptable they look like to you?
> 
> Looks not that bad. :-)
> But why do you need the file "include/cygwin/bits-byteswap.h"?
> 
> Artem, what do you think?
> IMHO adding cygwin support is less intrusive than adding native win32 support.

I would clean-up these patches even more, and split them further. But
yes, it looks like these could be turned into something not so intrusive
and maintainable and be merged if MTD users need them.

-- 
Best Regards,
Artem Bityutskiy