[U-Boot] [PATCH v2] Enable journal replay for UBIFS

[Heiko Schocher <hs@denx.de>]

Hello Anton,

Am 19.01.2015 14:48, schrieb Anton Habegger:
> During mount_ubifs the ubifs_replay_journal was disabled. This patch
> enables it again and fix some unrecoverable UBIFS volumes.
>
> The following patch enables the error handling for ubifs_replay_journal as well.
> Please see discussion "Unreadable UBIFS partition after power cuts".
>
>
> Signed-off-by: Anton Habegger <anton.habegger@delta-es.com>
> ---
>   fs/ubifs/Makefile |   2 +-
>   fs/ubifs/gc.c     | 987 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   fs/ubifs/replay.c |   8 +-
>   fs/ubifs/super.c  |   8 +-
>   fs/ubifs/tnc.c    |   7 +-
>   5 files changed, 997 insertions(+), 15 deletions(-)
>   create mode 100644 fs/ubifs/gc.c

Thanks!

> diff --git a/fs/ubifs/Makefile b/fs/ubifs/Makefile
> index 8c8c6ac..5efb349 100644
> --- a/fs/ubifs/Makefile
> +++ b/fs/ubifs/Makefile
> @@ -12,4 +12,4 @@
>   obj-y := ubifs.o io.o super.o sb.o master.o lpt.o
>   obj-y += lpt_commit.o scan.o lprops.o
>   obj-y += tnc.o tnc_misc.o debug.o crc16.o budget.o
> -obj-y += log.o orphan.o recovery.o replay.o
> +obj-y += log.o orphan.o recovery.o replay.o gc.o

Hmm.... do we really need this in u-boot? Ah, the most functions
are unneeded ... ok.

> diff --git a/fs/ubifs/gc.c b/fs/ubifs/gc.c
> new file mode 100644
> index 0000000..159ee67
> --- /dev/null
> +++ b/fs/ubifs/gc.c
> @@ -0,0 +1,987 @@
> +/*
> + * This file is part of UBIFS.
> + *
> + * Copyright (C) 2006-2008 Nokia Corporation.
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License version 2 as published by
> + * the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; if not, write to the Free Software Foundation, Inc., 51
> + * Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> + *
> + * Authors: Adrian Hunter
> + *          Artem Bityutskiy (???????? ?????)
> + */
> +
> +/*
> + * This file implements garbage collection. The procedure for garbage
> collection
> + * is different depending on whether a LEB as an index LEB (contains index
> + * nodes) or not. For non-index LEBs, garbage collection finds a LEB which
> + * contains a lot of dirty space (obsolete nodes), and copies the non-obsolete
> + * nodes to the journal, at which point the garbage-collected LEB is free to be
> + * reused. For index LEBs, garbage collection marks the non-obsolete
> index nodes
> + * dirty in the TNC, and after the next commit, the garbage-collected LEB is
> + * to be reused. Garbage collection will cause the number of dirty index nodes
> + * to grow, however sufficient space is reserved for the index to ensure the
> + * commit will never run out of space.
> + *
> + * Notes about dead watermark. At current UBIFS implementation we assume that
> + * LEBs which have less than @c->dead_wm bytes of free + dirty space are full
> + * and not worth garbage-collecting. The dead watermark is one min. I/O unit
> + * size, or min. UBIFS node size, depending on what is greater. Indeed, UBIFS
> + * Garbage Collector has to synchronize the GC head's write buffer before
> + * returning, so this is about wasting one min. I/O unit. However, UBIFS GC can
> + * actually reclaim even very small pieces of dirty space by garbage collecting
> + * enough dirty LEBs, but we do not bother doing this at this implementation.
> + *
> + * Notes about dark watermark. The results of GC work depends on how big are
> + * the UBIFS nodes GC deals with. Large nodes make GC waste more space. Indeed,
> + * if GC move data from LEB A to LEB B and nodes in LEB A are large, GC would
> + * have to waste large pieces of free space at the end of LEB B, because nodes
> + * from LEB A would not fit. And the worst situation is when all nodes are of
> + * maximum size. So dark watermark is the amount of free + dirty space in LEB
> + * which are guaranteed to be reclaimable. If LEB has less space, the GC might
> + * be unable to reclaim it. So, LEBs with free + dirty greater than dark
> + * watermark are "good" LEBs from GC's point of few. The other LEBs are not so
> + * good, and GC takes extra care when moving them.
> + */
> +#ifndef __UBOOT__
> +#include <linux/slab.h>
> +#include <linux/pagemap.h>
> +#include <linux/list_sort.h>
> +#endif
> +#include "ubifs.h"
> +
> +#ifndef __UBOOT__
> +/*
> + * GC may need to move more than one LEB to make progress. The below constants
> + * define "soft" and "hard" limits on the number of LEBs the garbage collector
> + * may move.
> + */
> +#define SOFT_LEBS_LIMIT 4
> +#define HARD_LEBS_LIMIT 32
> +
> +/**
> + * switch_gc_head - switch the garbage collection journal head.
> + * @c: UBIFS file-system description object
> + * @buf: buffer to write
> + * @len: length of the buffer to write
> + * @lnum: LEB number written is returned here
> + * @offs: offset written is returned here
> + *
> + * This function switch the GC head to the next LEB which is reserved in
> + * @c->gc_lnum. Returns %0 in case of success, %-EAGAIN if commit is required,
> + * and other negative error code in case of failures.
> + */
> +static int switch_gc_head(struct ubifs_info *c)
> +{
> + int err, gc_lnum = c->gc_lnum;
> + struct ubifs_wbuf *wbuf = &c->jheads[GCHD].wbuf;
> +
> + ubifs_assert(gc_lnum != -1);

Hmm.. something seems wrong with your mailer ... your patch is malformed.
Could you fix this please?

[...]
> diff --git a/fs/ubifs/replay.c b/fs/ubifs/replay.c
> index 7268b37..75b92ac 100644
> --- a/fs/ubifs/replay.c
> +++ b/fs/ubifs/replay.c
[...]
> @@ -1050,7 +1047,11 @@ int ubifs_replay_journal(struct ubifs_info *c)
>    * depend on it. This means we have to initialize it to make sure
>    * budgeting works properly.
>    */
> +#ifndef __UBOOT__
>    c->bi.uncommitted_idx = atomic_long_read(&c->dirty_zn_cnt);
> +#else
> + c->bi.uncommitted_idx = c->dirty_zn_cnt;
> +#endif

please introduce atomic_long_read() as mentioned to your other EMail.

>    c->bi.uncommitted_idx *= c->max_idx_node_sz;
>
>    ubifs_assert(c->bud_bytes <= c->max_bud_bytes || c->need_recovery);
> @@ -1063,4 +1064,3 @@ out:
>    c->replaying = 0;
>    return err;
>   }
> -#endif
> diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
> index 01d449a..10f8fff 100644
> --- a/fs/ubifs/super.c
> +++ b/fs/ubifs/super.c
> @@ -1049,7 +1049,6 @@ static void free_orphans(struct ubifs_info *c)
>    c->orph_buf = NULL;
>   }
>
> -#ifndef __UBOOT__
>   /**
>    * free_buds - free per-bud objects.
>    * @c: UBIFS file-system description object
> @@ -1061,7 +1060,6 @@ static void free_buds(struct ubifs_info *c)
>    rbtree_postorder_for_each_entry_safe(bud, n, &c->buds, rb)
>    kfree(bud);
>   }
> -#endif
>
>   /**
>    * check_volume_empty - check if the UBI volume is empty.
> @@ -1242,6 +1240,7 @@ static int ubifs_parse_options(struct ubifs_info
> *c, char *options,
>
>    return 0;
>   }
> +#endif
>
>   /**
>    * destroy_journal - destroy journal data structures.
> @@ -1272,7 +1271,6 @@ static void destroy_journal(struct ubifs_info *c)
>    ubifs_tnc_close(c);
>    free_buds(c);
>   }
> -#endif
>
>   /**
>    * bu_init - initialize bulk-read information.
> @@ -1502,11 +1500,9 @@ static int mount_ubifs(struct ubifs_info *c)
>    if (err)
>    goto out_lpt;
>
> -#ifndef __UBOOT__
>    err = ubifs_replay_journal(c);
>    if (err)
>    goto out_journal;
> -#endif
>
>    /* Calculate 'min_idx_lebs' after journal replay */
>    c->bi.min_idx_lebs = ubifs_calc_min_idx_lebs(c);
> @@ -1678,10 +1674,8 @@ out_infos:
>    spin_unlock(&ubifs_infos_lock);
>   out_orphans:
>    free_orphans(c);
> -#ifndef __UBOOT__
>   out_journal:
>    destroy_journal(c);
> -#endif
>   out_lpt:
>    ubifs_lpt_free(c, 0);
>   out_master:
> diff --git a/fs/ubifs/tnc.c b/fs/ubifs/tnc.c
> index 95cae54..b6737db 100644
> --- a/fs/ubifs/tnc.c
> +++ b/fs/ubifs/tnc.c
> @@ -2827,7 +2827,6 @@ out_unlock:
>    return ERR_PTR(err);
>   }
>
> -#ifndef __UBOOT__
>   /**
>    * tnc_destroy_cnext - destroy left-over obsolete znodes from a failed commit.
>    * @c: UBIFS file-system description object
> @@ -2850,7 +2849,6 @@ static void tnc_destroy_cnext(struct ubifs_info *c)
>    kfree(znode);
>    } while (cnext && cnext != c->cnext);
>   }
> -
>   /**
>    * ubifs_tnc_close - close TNC subsystem and free all related resources.
>    * @c: UBIFS file-system description object
> @@ -2859,17 +2857,20 @@ void ubifs_tnc_close(struct ubifs_info *c)
>   {
>    tnc_destroy_cnext(c);
>    if (c->zroot.znode) {
> +#ifndef __UBOOT__
>    long n;
> +#endif
>
>    ubifs_destroy_tnc_subtree(c->zroot.znode);
> +#ifndef __UBOOT__
>    n = atomic_long_read(&c->clean_zn_cnt);
>    atomic_long_sub(n, &ubifs_clean_zn_cnt);
> +#endif
>    }
>    kfree(c->gap_lebs);
>    kfree(c->ilebs);
>    destroy_old_idx(c);
>   }
> -#endif
>
>   /**
>    * left_znode - get the znode to the left.
>

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany