Re: [PATCH 0/4] replace genext2fs with populate-extfs.sh

[Jonathan Liu <net147@gmail.com>]

On 18/07/2013 10:32 PM, Robert Yang wrote:
>
> On 07/06/2013 10:49 PM, Jonathan Liu wrote:
>> Any updates?
>>
>
> Hi Jonathan,
>
> I've updated the patches, now they are working well, the "fsck -fn" finds
> no errors any more, but I have to send the patches to the ext4 mailing 
> list
> firstly, you can try it if you are interested in it atm.:
>
>   git://git.pokylinux.org/poky-contrib robert/image
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/image
>
>
> Robert Yang (5):
>   e2fsprogs: the max length of debugfs argument is too short
>   e2fsprogs: let debugfs do sparse copy
>   e2fsprogs: only update the icache for ext2_inode
>   e2fsprogs: add populate-extfs.sh
>   image_types.bbclass: replace genext2fs with populate-extfs.sh
>
>  meta/classes/image_types.bbclass                   |  46 +++---
>  .../e2fsprogs-1.42.8/debugfs-too-short.patch       |  28 ++++
>  .../e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch    |  69 ++++++++
>  .../e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh   |  96 +++++++++++
>  .../e2fsprogs/e2fsprogs-1.42.8/sparse_copy.patch   | 177 
> +++++++++++++++++++++
>  .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.8.bb |   5 +
>  6 files changed, 393 insertions(+), 28 deletions(-)
>  create mode 100644 
> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/debugfs-too-short.patch
>  create mode 100644 
> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/fix-icache.patch
>  create mode 100644 
> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/populate-extfs.sh
>  create mode 100644 
> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.8/sparse_copy.patch
>
> // Robert
Thanks, I will test it tomorrow. "yes | mkfs.$fstype" could probably be 
replaced with "mkfs.$fstype -F".

Regards,
Jonathan
>
>> Regards,
>> Jonathan
>>>
>>>> meta: f7afeeb75993b159bb8959e0309bc5eb3978a8fb
>>>>
>>>> conf/local.conf:
>>>> BB_NUMBER_THREADS = "8"
>>>> PARALLEL_MAKE = "-j 8"
>>>> MACHINE ??= "qemux86"
>>>> DISTRO ?= "poky"
>>>> PACKAGE_CLASSES = "package_ipk"
>>>> EXTRA_IMAGE_FEATURES = "debug-tweaks"
>>>> USER_CLASSES ?= "buildstats image-mklibs image-prelink"
>>>> PATCHRESOLVE = "noop"
>>>> BB_DISKMON_DIRS = "\
>>>>      STOPTASKS,${TMPDIR},1G,100K \
>>>>      STOPTASKS,${DL_DIR},1G,100K \
>>>>      STOPTASKS,${SSTATE_DIR},1G,100K \
>>>>      ABORT,${TMPDIR},100M,1K \
>>>>      ABORT,${DL_DIR},100M,1K \
>>>>      ABORT,${SSTATE_DIR},100M,1K"
>>>> CONF_VERSION = "1"
>>>> DISTRO_FEATURES_append = " systemd"
>>>> DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
>>>> VIRTUAL-RUNTIME_init_manager = "systemd"
>>>> IMAGE_FSTYPES = "ext4"
>>>>
>>>> # bitbake core-image-minimal
>>>> # fsck.ext4 -fn tmp/deploy/images/core-image-minimal-qemux86.ext4
>>>>
>>>> Regards,
>>>> Jonathan
>>>>>
>>>>>> distro conf:
>>>>>> DISTRO_FEATURES_append = " largefile opengl systemd"
>>>>>> DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
>>>>>> VIRTUAL-RUNTIME_graphical_init_manager = ""
>>>>>> VIRTUAL-RUNTIME_init_manager = "systemd"
>>>>>> PACKAGE_CLASSES = "package_ipk"
>>>>>>
>>>>>> image bb:
>>>>>> SYSLINUX_ROOT ?= "root=/dev/sda2"
>>>>>> SYSLINUX_PROMPT = "0"
>>>>>> SYSLINUX_TIMEOUT = "1"
>>>>>> SYSLINUX_LABELS = "boot"
>>>>>> LABELS_append = " ${SYSLINUX_LABELS} "
>>>>>> do_bootdirectdisk[depends] += "${PN}:do_rootfs"
>>>>>> APPEND += "rootfstype=ext4 rw"
>>>>>> ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_BASENAME}-${MACHINE}.ext4"
>>>>>> inherit boot-directdisk
>>>>>> IMAGE_FEATURES += "hwcodecs package-management ssh-server-openssh 
>>>>>> x11"
>>>>>> IMAGE_FSTYPES = "ext4"
>>>>>> IMAGE_LINGUAS = ""
>>>>>>
>>>>>>>
>>>>>>> My configuration:
>>>>>>> MACHINE = "qemux86"
>>>>>>> IMAGE_FSTYPES += "ext4"
>>>>>>>
>>>>>>> // Robert
>>>>>>>
>>>>>>> On 05/13/2013 12:59 PM, Jonathan Liu wrote:
>>>>>>>> Hi Robert,
>>>>>>>>
>>>>>>>> I get the following errors doing filesystem check on the ext4 
>>>>>>>> image:
>>>>>>>>
>>>>>>>> # fsck.ext4 -fn tmp/deploy/images/custom-image-custom.ext4.ext4
>>>>>>>> e2fsck 1.42.7 (21-Jan-2013)
>>>>>>>> Pass 1: Checking inodes, blocks, and sizes
>>>>>>>> Pass 2: Checking directory structure
>>>>>>>> Entry 'mtd7' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd4' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'fb0' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd5' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty6' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'random' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be
>>>>>>>> 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd1' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'urandom' in /dev (2739) has an incorrect filetype (was 
>>>>>>>> 4, should be
>>>>>>>> 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'zero' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'null' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'kmsg' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'kmem' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd6' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty7' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty4' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'apm_bios' in /dev (2739) has an incorrect filetype (was 
>>>>>>>> 4, should be
>>>>>>>> 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty5' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd0' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty0' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'ttySA0' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be
>>>>>>>> 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mem' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd2' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty8' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'console' in /dev (2739) has an incorrect filetype (was 
>>>>>>>> 4, should be
>>>>>>>> 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'ttyS0' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty2' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'mtd3' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty1' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Entry 'tty3' in /dev (2739) has an incorrect filetype (was 4, 
>>>>>>>> should be 3).
>>>>>>>> Fix? no
>>>>>>>>
>>>>>>>> Pass 3: Checking directory connectivity
>>>>>>>> Pass 4: Checking reference counts
>>>>>>>> Inode 774 ref count is 9, should be 10.  Fix? no
>>>>>>>>
>>>>>>>> Pass 5: Checking group summary information
>>>>>>>>
>>>>>>>> tmp/deploy/images/custom-image-custom.ext4.ext4: ********** 
>>>>>>>> WARNING:
>>>>>>>> Filesystem still has errors **********
>>>>>>>>
>>>>>>>> tmp/deploy/images/custom-image-custom.ext4.ext4: 3899/21336 
>>>>>>>> files (0.5%
>>>>>>>> non-contiguous), 131042/169656 blocks
>>>>>>>>
>>>>>>>>
>>>>>>>> After booting the image I get the following:
>>>>>>>> # dmesg | grep EXT4-fs
>>>>>>>> EXT4-fs (sda2): mounted filesystem with ordered data mode. 
>>>>>>>> Opts: (null)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd: bad header/extent: invalid magic - magic 0, entries 0, 
>>>>>>>> max 0(0),
>>>>>>>> depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd: bad header/extent: invalid magic - magic 0, entries 0, 
>>>>>>>> max 0(0),
>>>>>>>> depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd: bad header/extent: invalid magic - magic 0, entries 0, 
>>>>>>>> max 0(0),
>>>>>>>> depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-machine: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-machine: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-journal: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs (sda2): re-mounted. Opts: (null)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-journal: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-journal: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-journal: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #883: comm
>>>>>>>> systemd-journal: bad header/extent: invalid magic - magic 0, 
>>>>>>>> entries 0, max
>>>>>>>> 0(0), depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #810: comm
>>>>>>>> login: bad header/extent: invalid magic - magic 0, entries 0, 
>>>>>>>> max 0(0),
>>>>>>>> depth 0(0)
>>>>>>>> EXT4-fs error (device sda2): ext4_ext_check_inode:467: inode 
>>>>>>>> #810: comm
>>>>>>>> sshd: bad header/extent: invalid magic - magic 0, entries 0, 
>>>>>>>> max 0(0),
>>>>>>>> depth 0(0)
>>>>>>>>
>>>>>>>>
>>>>>>>> # systemctl --failed | cat
>>>>>>>> UNIT                     LOAD   ACTIVE SUB DESCRIPTION
>>>>>>>> machineid.service        loaded failed failed Machine ID first 
>>>>>>>> boot
>>>>>>>> configure
>>>>>>>> systemd-journald.service loaded failed failed Journal Service
>>>>>>>> systemd-journald.socket  loaded failed failed Journal Socket
>>>>>>>>
>>>>>>>> LOAD   = Reflects whether the unit definition was properly loaded.
>>>>>>>> ACTIVE = The high-level unit activation state, i.e. 
>>>>>>>> generalization of SUB.
>>>>>>>> SUB    = The low-level unit activation state, values depend on 
>>>>>>>> unit type.
>>>>>>>>
>>>>>>>> 3 loaded units listed. Pass --all to see loaded but inactive 
>>>>>>>> units, too.
>>>>>>>> To show all installed unit files use 'systemctl list-unit-files'.
>>>>>>>>
>>>>>>>>
>>>>>>>> I am booting kernel with rootfstype=ext4. IMAGE_FSTYPES = "ext4".
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Jonathan
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7 May 2013 19:48, Robert Yang <liezhi.yang@windriver.com> 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> * The benefits:
>>>>>>>>>    - Really support ext4
>>>>>>>>>    - Support the sparse file (the sparse file became into the 
>>>>>>>>> common file
>>>>>>>>> before)
>>>>>>>>>    - Have a uniform code for ext2/3/4 generation
>>>>>>>>>    - Remove the depends on genext2fs-native
>>>>>>>>>
>>>>>>>>> * Impact
>>>>>>>>>    - Build time:
>>>>>>>>>      a) If we build fresh core-image-sato, there is nearly no 
>>>>>>>>> impact.
>>>>>>>>>      b) If we do the image generation, which means:
>>>>>>>>>         $ bitbake core-image-sato
>>>>>>>>>         $ bitbake core-image-sato -ccleansstate
>>>>>>>>>         $ bitbake core-image-sato
>>>>>>>>>     About 50 extra seconds are needed, here is my test result:
>>>>>>>>>     Before the patches: 4m25s
>>>>>>>>>     After the patches: 5m17s
>>>>>>>>>     This is because the genext2fs is much faster than the
>>>>>>>>>     populate-extfs.sh, we will replace this script by the 
>>>>>>>>> mke2fs when it
>>>>>>>>>     supports create the filesystem from a initial directory.
>>>>>>>>>
>>>>>>>>>    - Disk space (take core-image-sato as an example)
>>>>>>>>>      a) The image file size is the same as before (529M)
>>>>>>>>>      b) The disk usage is a little different: (du -sh)
>>>>>>>>>                before       now
>>>>>>>>>         ext2:  364M         388M
>>>>>>>>>         ext3:  381M         404M
>>>>>>>>>         ext4:  380M         387M
>>>>>>>>>
>>>>>>>>>     We may need to adjust the IMAGE_OVERHEAD_FACTOR from 1.3 
>>>>>>>>> to 1.4.
>>>>>>>>>
>>>>>>>>>     I have done some simple runtime testing on core-image-sato 
>>>>>>>>> and
>>>>>>>>>     core-image-minimal, they worked well.
>>>>>>>>>
>>>>>>>>> // Robert
>>>>>>>>>
>>>>>>>>> The following changes since commit
>>>>>>>>> 3472c1f7ab409cd91c1d4782d9e00880b84e3ae8:
>>>>>>>>>
>>>>>>>>>    grub-efi-native: Cleanup whitespace (2013-05-03 16:37:05 
>>>>>>>>> +0100)
>>>>>>>>>
>>>>>>>>> are available in the git repository at:
>>>>>>>>>
>>>>>>>>>    git://git.pokylinux.org/poky-contrib robert/ext4
>>>>>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/ext4
>>>>>>>>>
>>>>>>>>> Robert Yang (4):
>>>>>>>>>    e2fsprogs: the max length of debugfs argument is too short
>>>>>>>>>    e2fsprogs: let debugfs do sparse copy
>>>>>>>>>    e2fsprogs: add populate-extfs.sh
>>>>>>>>>    image_types.bbclass: replace genext2fs with populate-extfs.sh
>>>>>>>>>
>>>>>>>>> meta/classes/image_types.bbclass | 46 ++++----
>>>>>>>>> .../e2fsprogs-1.42.7/debugfs-too-short.patch | 28 +++++
>>>>>>>>> .../e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh | 93 
>>>>>>>>> ++++++++++++++++
>>>>>>>>> .../e2fsprogs/e2fsprogs-1.42.7/sparse_copy.patch | 114
>>>>>>>>> ++++++++++++++++++++
>>>>>>>>> .../recipes-devtools/e2fsprogs/e2fsprogs_1.42.7.bb | 4 +
>>>>>>>>>   5 files changed, 257 insertions(+), 28 deletions(-)
>>>>>>>>>   create mode 100644
>>>>>>>>> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/debugfs-too-short.patch 
>>>>>>>>>
>>>>>>>>>   create mode 100644
>>>>>>>>> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/populate-extfs.sh 
>>>>>>>>>
>>>>>>>>>   create mode 100644
>>>>>>>>> meta/recipes-devtools/e2fsprogs/e2fsprogs-1.42.7/sparse_copy.patch 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> 1.7.10.4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Openembedded-core mailing list
>>>>>>>>> Openembedded-core@lists.openembedded.org
>>>>>>>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>> Regards,
>>>>>> Jonathan
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>
>>
>>