Shrink UBIFS

Shrink UBIFS

[Ryan Meulenkamp]

Hi,


I'm planning to write an ioctl for shrinking a UBIFS to be able to resize the volume its on and create another

volume because it is essential for our upgrade/migration flow. Do you guys have any advice for me? The

hard part is that LEB's that would fall outside the new size should be moved inside the new size. From what

I read, the garbage collection code could be used to  accomplish this, but it is not really possible to define

which LEB's to put where.


Thanks in advance!


Ryan

Re: Shrink UBIFS

[Richard Weinberger]

Ryan,

On Thu, Sep 28, 2017 at 1:43 PM, Ryan Meulenkamp
<Ryan.Meulenkamp@nedap.com> wrote:
> Hi,
>
>
> I'm planning to write an ioctl for shrinking a UBIFS to be able to resize the volume its on and create another
>
> volume because it is essential for our upgrade/migration flow. Do you guys have any advice for me? The
>
> hard part is that LEB's that would fall outside the new size should be moved inside the new size. From what
>
> I read, the garbage collection code could be used to  accomplish this, but it is not really possible to define
>
> which LEB's to put where.
>
>
> Thanks in advance!

So, you want to implement _online_ shrinking?
This is a way more complicated, think of power-cuts.
I strongly suggest thinking about offline shrinking first.

You are right, the garbage collector might be useful, you could modify
it in a way
to move blocks away from to-be-removed LEBs.
But again, do you *really* need online shrinking?

-- 
Thanks,
//richard

Re: Shrink UBIFS

[Artem Bityutskiy]

On Sat, 2017-09-30 at 20:09 +0200, Richard Weinberger wrote:
> Ryan,
> 
> On Thu, Sep 28, 2017 at 1:43 PM, Ryan Meulenkamp
> <Ryan.Meulenkamp@nedap.com> wrote:
> > Hi,
> > 
> > 
> > I'm planning to write an ioctl for shrinking a UBIFS to be able to
> > resize the volume its on and create another
> > 
> > volume because it is essential for our upgrade/migration flow. Do
> > you guys have any advice for me? The
> > 
> > hard part is that LEB's that would fall outside the new size should
> > be moved inside the new size. From what
> > 
> > I read, the garbage collection code could be used to  accomplish
> > this, but it is not really possible to define
> > 
> > which LEB's to put where.
> > 
> > 
> > Thanks in advance!
> 
> So, you want to implement _online_ shrinking?
> This is a way more complicated, think of power-cuts.
> I strongly suggest thinking about offline shrinking first.
> 
> You are right, the garbage collector might be useful, you could
> modify
> it in a way
> to move blocks away from to-be-removed LEBs.
> But again, do you *really* need online shrinking?

Yeah, offline would be way easier


But still just thinking aloud about online shrinking...

Suppose we have UBIFS which looks like this.

FDDDEFDFDFFFFFDDDFDFFEEEDFDDDFDFDFDFDFDDDDDDFFFFFFDDDDDDDDDDDFFDFF

F - full eraseblock
D - eraseblock with dirty space.
E - empty eraseblock.

To shrink, we need to turn it into something like this.

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFDDDDEEEEEEEEEEEEEEEEEEEE

Currently GC selects the dirtiest eraseblock as the victim for
cleaning, because this is the most efficient strategy. But you could
introduce a special 'shrinking  mode', where you'd tell GC to try hard
picking the victims from the end, Probably also try hard not to re-use
the empty eraseblocks at the end.

Then while being in this mode, GC is used from a background thread to
make enough empty eraseblocks at the end. Well, it is possible that GC
will stop making progress before enough eraseblocks at the end are made
empty, which would mean that further shrinking is impossible.

Just thoughts.

Re: Shrink UBIFS

[Richard Weinberger]

Am Mittwoch, 4. Oktober 2017, 14:23:56 CEST schrieb Ryan Meulenkamp:
> So, I stranded on the following: after performing the resize action,
> I get a nice strain of errors telling me that the node type is bad:
> (255 but expected <1 to 9>). I suspect this is because the block is
> erased but UBIFS did not expect it to be. Is this a problem with
> garbage collection of the index nodes? I don't understand how
> index nodes are garbage collected, and where they end up.

How can we know? :-)

But yes, seems like an UBIFS structure points to an erased region.

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y

Re: Shrink UBIFS

[Richard Weinberger]

Ryan,

Am Donnerstag, 19. Oktober 2017, 09:13:03 CET schrieb Ryan Meulenkamp:
> Hi,
> 
> I've been working on something else for a while, but this is the status on
> resizing so far:

A diff is not really a good status report. ;)

Can you please outline what works so far and what not?
Is there some point where you need technical input from us?

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y

Re: Shrink UBIFS

[Artem Bityutskiy]

On Mon, 2017-11-06 at 22:27 +0100, Richard Weinberger wrote:
> Ryan,
> 
> Am Donnerstag, 19. Oktober 2017, 09:13:03 CET schrieb Ryan
> Meulenkamp:
> > Hi,
> > 
> > I've been working on something else for a while, but this is the
> > status on
> > resizing so far:
> 
> A diff is not really a good status report. ;)
> 
> Can you please outline what works so far and what not?
> Is there some point where you need technical input from us?

Yeah, it would be helpful if you could invest some time writing a good
overview. What's the goal, what's the strategy achieving it, what has
been tried, what worked, what did not work, what are the main
obstacles, what was achieved, what are the tradeoffs, etc.