GC operation

GC operation

[Bernhard Priewasser]

Hello,

a question just for better understanding of JFFS2...

When and how is GC called?
jffs2_garbage_collect_pass() is called from GC thread AND after each 
writing to flash. But does a single call of jffs2_garbage_collect_pass() 
(or jffs2_garbage_collect_trigger() or SIGHUP) proceed only one node at 
a time?  For each node / for each block / ..... some loop?

Ah, another one: Are blocks on the erase_pending_list erased beside the 
GC cycles?

Hm, sorry for the annoying questions, but garbage collection can be an 
ugly and confusing thing if one is not used to it :-) Thanks for your 
assistance!

-- 
Bernhard

Re: GC operation

[Artem B. Bityutskiy]

Bernhard Priewasser wrote:
> When and how is GC called?
 From the GC thread and when there is no (or few) free space to write. 
In the latter case the writing process is blocked and waits until GC has 
freed some space.

> (or jffs2_garbage_collect_trigger() or SIGHUP) proceed only one node at 
> a time?  For each node / for each block / ..... some loop?
jffs2_garbage_collect_pass() moves one node at a time.

> Ah, another one: Are blocks on the erase_pending_list erased beside the 
> GC cycles?
They are erased later, yes.

> Hm, sorry for the annoying questions, but garbage collection can be an 
> ugly and confusing thing if one is not used to it :-)
Brr, didn't get it.. GC may be ugly if what?

-- 
Best Regards,
Artem B. Bityutskiy,
St.-Petersburg, Russia.

Re: GC operation

[Bernhard Priewasser]

Hi Artem,

 > Brr, didn't get it.. GC may be ugly if what?
If someone wants to understand how it works in detail :-)

 >> When and how is GC called?
 > From the GC thread and when there is no (or few) free space to write.
 > In the latter case the writing process is blocked and waits until GC
 > has freed some space.
E.g. if it is considered as neccessary either by jffs2_reserve_space()
or jffs2_thread_should_wake().
Something about the blocking topic... If there is almost no free space
and a write command issued, can it be blocked until the whole partition
is GC'd (worst case)?? What a latency time... Are there mechanisms to
avoid/control this? What about the "erase suspend" thing?

 >> Ah, another one: Are blocks on the erase_pending_list erased beside
 >> the GC cycles?
 > They are erased later, yes.
jffs2_erase_pending_blocks(), am I right? When is it called? I can only
find it in jffs2_write_super() with count=0 and jffs2_find_nextblock()
with count=1.

Many thanks,
Bernhard

Re: GC operation

[Artem B. Bityutskiy]

Hi Bernhard,

Bernhard Priewasser wrote:
>  > Brr, didn't get it.. GC may be ugly if what?
> If someone wants to understand how it works in detail :-)
So, the problem is that you think the code is not understandable? Offer 
your changes then.

> 
>  >> When and how is GC called?
>  > From the GC thread and when there is no (or few) free space to write.
>  > In the latter case the writing process is blocked and waits until GC
>  > has freed some space.
> E.g. if it is considered as neccessary either by jffs2_reserve_space()
> or jffs2_thread_should_wake().
> Something about the blocking topic... If there is almost no free space
> and a write command issued, can it be blocked until the whole partition
> is GC'd (worst case)?? What a latency time... Are there mechanisms to
> avoid/control this? What about the "erase suspend" thing?
Hmm.

Suppose we have an almost full partition. You write N bytes. There is 
only M free space, M < N. If there is N-M of dirty space, GC is started. 
Otherwise, -ENOSPC. GC will work until there is enough space to write. 
No need to GC the whole partition... Only if the dirty space is 
distributed over it, then yes. And yes, the more data is on the FS, the 
less is the GC performance and the greater is the average latency...

Or, since blocks to GC are picked randomly, we may end up with GC of the 
whole partition, hypothetically, but unlikely.

> jffs2_erase_pending_blocks(), am I right? When is it called? I can only
> find it in jffs2_write_super() with count=0 and jffs2_find_nextblock()
> with count=1.
So, you've answered your question :-)

HTH,
Artem.

-- 
Best Regards,
Artem B. Bityutskiy,
St.-Petersburg, Russia.

Re: GC operation

[Bernhard Priewasser]

Hi Artem,

> So, the problem is that you think the code is not understandable? Offer 
> your changes then.
No, the problem is that it's not always understandable for ME, that's a 
big difference :-)

> Suppose we have an almost full partition. You write N bytes. There is 
> only M free space, M < N. If there is N-M of dirty space, GC is started. 
> Otherwise, -ENOSPC. GC will work until there is enough space to write. 
> No need to GC the whole partition... Only if the dirty space is 
> distributed over it, then yes. And yes, the more data is on the FS, the 
> less is the GC performance and the greater is the average latency...
> 
> Or, since blocks to GC are picked randomly, we may end up with GC of the 
> whole partition, hypothetically, but unlikely.
OK. Things become more clearly. A write request to an almost full 
partition always issues garbage collecting just as much space (nodes) 
which is neccessary for writing the new data? And full GC is performed 
only at mount? (And if c->resv_blocks_gctrigger is reached?)

>> jffs2_erase_pending_blocks(), am I right? When is it called? I can only
>> find it in jffs2_write_super() with count=0 and jffs2_find_nextblock()
>> with count=1.
> So, you've answered your question :-)
I meant the way it is called by kupdated (which I don't know at all).

> HTH,
It did! Thank you for patience.

-- 
Bernhard

Re: GC operation

[Josh Boyer]

On 11/9/05, Bernhard Priewasser <priewasser@gmail.com> wrote:
>
> >> jffs2_erase_pending_blocks(), am I right? When is it called? I can only
> >> find it in jffs2_write_super() with count=0 and jffs2_find_nextblock()
> >> with count=1.
> > So, you've answered your question :-)
> I meant the way it is called by kupdated (which I don't know at all).

The kupdated thread wakes up every 5 seconds or so and calls
sync_supers, which will eventually call jffs2_write_super() and hence
jffs2_erase_pending_blocks is called.

josh

Re: GC operation

[Artem B. Bityutskiy]

Bernhard Priewasser wrote:
>> Suppose we have an almost full partition. You write N bytes. There is 
>> only M free space, M < N. If there is N-M of dirty space, GC is 
>> started. Otherwise, -ENOSPC. GC will work until there is enough space 
>> to write. No need to GC the whole partition... Only if the dirty space 
>> is distributed over it, then yes. And yes, the more data is on the FS, 
>> the less is the GC performance and the greater is the average latency...
>>
>> Or, since blocks to GC are picked randomly, we may end up with GC of 
>> the whole partition, hypothetically, but unlikely.
> 
> OK. Things become more clearly. A write request to an almost full 
> partition always issues garbage collecting just as much space (nodes) 
> which is neccessary for writing the new data? And full GC is performed 
> only at mount? (And if c->resv_blocks_gctrigger is reached?)
Not almost full partition. Here are some statements:

1. GC starts if there are no *free eraseblocks*.
2. GC makes free space in units of an eraseblock.
3. No GC is needed on mount.

JFFS2 does *checking* on mount, i.e., checks all CRCs. It is done in the 
context of the GC thread, but does not actually relate to the Garbage 
Collection. This is just how it is implemented.

>>> jffs2_erase_pending_blocks(), am I right? When is it called? I can only
>>> find it in jffs2_write_super() with count=0 and jffs2_find_nextblock()
>>> with count=1.
>>
>> So, you've answered your question :-)
> 
> I meant the way it is called by kupdated (which I don't know at all).
Will just add to what Josh said.

jffs2_write_super() is called when the JFFS2 superblock is dirty 
(JFFS2's sb->s_dirt is set). JFFS2 marks its "struct superblock" 
structure as dirty when something is added to the erase_pending_list or 
written to the wbuf.

When sb->s_dirt, kupdated will call jffs2_write_super() in some 
configurable time and JFFS2 will erase pending eraseblocks and flush the 
wbuf.

I'm not sure why JFFS2 does not use its own timer for this. But it seems 
it used to.

-- 
Best Regards,
Artem B. Bityutskiy,
St.-Petersburg, Russia.

Re: GC operation

[Bernhard Priewasser]

> Not almost full partition. Here are some statements:
> 
> 1. GC starts if there are no *free eraseblocks*.
> 2. GC makes free space in units of an eraseblock.
> 3. No GC is needed on mount.
> 
> JFFS2 does *checking* on mount, i.e., checks all CRCs. It is done in the 
> context of the GC thread, but does not actually relate to the Garbage 
> Collection. This is just how it is implemented.
Blocking time is rather important for us, e.g. the GC and erase_pending 
stuff is too. Thank you for bringing some light into it, perhaps it 
wouldn't be bad if there was some documentation. I'm doing my Master's 
Thesis right now on JFFS2, that might be a good documention.

> jffs2_write_super() is called when the JFFS2 superblock is dirty 
> (JFFS2's sb->s_dirt is set). JFFS2 marks its "struct superblock" 
> structure as dirty when something is added to the erase_pending_list or 
> written to the wbuf.
> 
> When sb->s_dirt, kupdated will call jffs2_write_super() in some 
> configurable time and JFFS2 will erase pending eraseblocks and flush the 
> wbuf.
If anybody else is intersted: See 
http://lists.infradead.org/pipermail/linux-mtd/2005-November/014271.html
and follow-up.

-- 
Bernhard