e2fsck: do not skip deeper checkers when s_last_orphan list has truncated inodes

e2fsck: do not skip deeper checkers when s_last_orphan list has truncated inodes

[zhanchengbin]

If the system crashes when a file is being truncated, we will get a 
problematic inode,
and it will be added into fs->super->s_last_orphan.
When we run `e2fsck -a img`, the s_last_orphan list will be traversed 
and deleted.
During this period, orphan inodes in the s_last_orphan list with 
i_links_count==0 can
be deleted, and orphan inodes with  i_links_count !=0 (ex. the truncated 
inode)
cannot be deleted. However, when there are some orphan inodes with 
i_links_count !=0,
the EXT2_VALID_FS is still assigned to fs->super->s_state, the deeper 
checkers are skipped
with some inconsistency problems.
Here, we will clean EXT2_VALID_FS flag when there is orphan inodes with 
i_links_count !=0
for deeper checkers.

Problems with truncated files.
     [root@localhost ~]# e2fsck -a img
     img: recovering journal
     img: Truncating orphaned inode 188 (uid=0, gid=0, mode=0100666, size=0)
     img: Truncating orphaned inode 174 (uid=0, gid=0, mode=0100666, size=0)
     img: clean, 484/128016 files, 118274/512000 blocks
     [root@localhost ~]# e2fsck -fn img
     e2fsck 1.46.5 (30-Dec-2021)
     Pass 1: Checking inodes, blocks, and sizes
     Inode 174, i_blocks is 2, should be 0.  Fix? no

     Inode 188, i_blocks is 2, should be 0.  Fix? no

     Pass 2: Checking directory structure
     Pass 3: Checking directory connectivity
     Pass 4: Checking reference counts
     Pass 5: Checking group summary information

     img: ********** WARNING: Filesystem still has errors **********

     img: 484/128016 files (24.6% non-contiguous), 118274/512000 blocks
     [root@localhost ~]# e2fsck -a img
     img: clean, 484/128016 files, 118274/512000 blocks

But, if run `e2fsck -f img`, EXT2_VALID_FS flag will be clean, so do 
`e2fsck -a img` again,
can fix this problem.

     [root@localhost ~]# e2fsck -f img
     e2fsck 1.46.5 (30-Dec-2021)
     Pass 1: Checking inodes, blocks, and sizes
     Inode 174, i_blocks is 2, should be 0.  Fix<y>? no
     Inode 188, i_blocks is 2, should be 0.  Fix<y>? no
     Pass 2: Checking directory structure
     Pass 3: Checking directory connectivity
     Pass 4: Checking reference counts
     Pass 5: Checking group summary information

     img: ********** WARNING: Filesystem still has errors **********

     img: 484/128016 files (24.6% non-contiguous), 118274/512000 blocks
     [root@localhost ~]# e2fsck -a img
     img was not cleanly unmounted, check forced.
     img: Inode 174, i_blocks is 2, should be 0.  FIXED.
     img: Inode 188, i_blocks is 2, should be 0.  FIXED.
     img: 484/128016 files (24.6% non-contiguous), 118274/512000 blocks

Signed-off-by: zhanchengbin <zhanchengbin1@huawei.com>
---
  e2fsck/super.c | 1 +
  1 file changed, 1 insertion(+)

diff --git a/e2fsck/super.c b/e2fsck/super.c
index 9495e029..f4a414b7 100644
--- a/e2fsck/super.c
+++ b/e2fsck/super.c
@@ -351,6 +351,7 @@ static int release_orphan_inode(e2fsck_t ctx, 
ext2_ino_t *ino, char *block_buf)
          inode.i_dtime = ctx->now;
      } else {
          inode.i_dtime = 0;
+        fs->super->s_state &= ~EXT2_VALID_FS;
      }
      e2fsck_write_inode_full(ctx, *ino, EXT2_INODE(&inode),
                  sizeof(inode), "delete_file");

Re: e2fsck: do not skip deeper checkers when s_last_orphan list has truncated inodes

[Theodore Ts'o]

On Tue, Mar 15, 2022 at 04:01:45PM +0800, zhanchengbin wrote:
> If the system crashes when a file is being truncated, we will get a
> problematic inode,
> and it will be added into fs->super->s_last_orphan.
> When we run `e2fsck -a img`, the s_last_orphan list will be traversed and
> deleted.
> During this period, orphan inodes in the s_last_orphan list with
> i_links_count==0 can
> be deleted, and orphan inodes with  i_links_count !=0 (ex. the truncated
> inode)
> cannot be deleted. However, when there are some orphan inodes with
> i_links_count !=0,
> the EXT2_VALID_FS is still assigned to fs->super->s_state, the deeper
> checkers are skipped
> with some inconsistency problems.

That's not supposed to happen.  We regularly put inodes on the orphan
list when they are being truncated so that if we crash, the truncation
operation can be completed as part of the journal recovery and remount
operation.  This is true regardles sof whether the recovery is done by
e2fsck or by the kernel.

If a crash during a truncate leads to an inconsistent file system
after the file system is mounted, or after e2fsck does the journal
replay and orphan inode list processing, that's a kernel bug, and we
should fix the bug in the kernel.

Do you have a reliable reproducer for this situation?

Thanks,

						- Ted

Re: e2fsck: do not skip deeper checkers when s_last_orphan list has truncated inodes

[zhanchengbin]

在 2022/3/16 1:54, Theodore Ts'o 写道:
> On Tue, Mar 15, 2022 at 04:01:45PM +0800, zhanchengbin wrote:
>> If the system crashes when a file is being truncated, we will get a
>> problematic inode,
>> and it will be added into fs->super->s_last_orphan.
>> When we run `e2fsck -a img`, the s_last_orphan list will be traversed and
>> deleted.
>> During this period, orphan inodes in the s_last_orphan list with
>> i_links_count==0 can
>> be deleted, and orphan inodes with  i_links_count !=0 (ex. the truncated
>> inode)
>> cannot be deleted. However, when there are some orphan inodes with
>> i_links_count !=0,
>> the EXT2_VALID_FS is still assigned to fs->super->s_state, the deeper
>> checkers are skipped
>> with some inconsistency problems.
> 
> That's not supposed to happen.  We regularly put inodes on the orphan
> list when they are being truncated so that if we crash, the truncation
> operation can be completed as part of the journal recovery and remount
> operation.  This is true regardles sof whether the recovery is done by
> e2fsck or by the kernel.

Yes, you are right.
Truncated has been completed，and file ACL has been set to zero in
release_inode_blocks(), but the i_blocks was not subtracted acl blocks.
So i_blocks is inconsistent。
Li Jinlin sent a patch yesterday to fix it.

> 
> If a crash during a truncate leads to an inconsistent file system
> after the file system is mounted, or after e2fsck does the journal
> replay and orphan inode list processing, that's a kernel bug, and we
> should fix the bug in the kernel.
> 
> Do you have a reliable reproducer for this situation?

I have a reproducer but it is not necessarily:
#!/bin/bash
disk_list=$(multipath -ll | grep filedisk | awk '{print $1}')

for disk in ${disk_list}
do
     mkfs.ext4 -F /dev/mapper/$disk
     mkdir ${disk}
done

function err_inject()
{
     iscsiadm -m node -p 127.0.0.1 -u &> /dev/null
     iscsiadm -m node -p 127.0.0.1 -l &> /dev/null
     sleep 1
     iscsiadm -m node -p 9.82.236.206 -u &> /dev/null
     iscsiadm -m node -p 9.82.236.206 -l &> /dev/null
     sleep 1

     iscsiadm -m node -p 127.0.0.1 -u &> /dev/null
     iscsiadm -m node -p 127.0.0.1 -l &> /dev/null
     iscsiadm -m node -p 9.82.236.206 -u &> /dev/null
     iscsiadm -m node -p 9.82.236.206 -l &> /dev/null
     sleep 1
}



count=0
while true
do
     ((count=count+1))
     for disk in ${disk_list}
     do
         while true
         do
             mount -o data_err=abort,errors=remount-ro /dev/mapper/$disk 
$disk && break
             sleep 0.1
         done
         nohup fsstress -d $(pwd)/$disk -l 10 -n 1000 -p 10 &>/dev/null &
     done

     sleep 5

     for disk in ${disk_list}
     do
         dm=$(multipath -ll | grep -w $disk | awk '{print $2}')
         aqu_sz=$(iostat -x 1 -d 2 | grep -w $dm | tail -1 | awk '{print 
$(NF-1)}')
         util=$(iostat -x 1 -d 2 | grep -w $dm | tail -1 | awk '{print 
$NF}')
         #if [ "${aqu_sz}" == "0.00" -o "$util" == "0.00" ];then
         #    iostat -x 1 -d 2
         #    exit 1
         #fi
         mount | grep $disk | grep '(ro' && exit 1
     done

     err_inject

     while [ -n "`pidof fsstress`" ]
     do
         sleep 1
     done

     for disk in ${disk_list}
     do
         umount $disk
         dm=$(multipath -ll | grep -w $disk | awk '{print $2}')
         aqu_sz=$(iostat -x 1 -d 2 | grep -w $dm | tail -1 | awk '{print 
$(NF-1)}')
         util=$(iostat -x 1 -d 2 | grep -w $dm | tail -1 | awk '{print 
$NF}')
         if [ "${aqu_sz}" != "0.00" -o "$util" != "0.00" ];then
             iostat -x 1 -d 2
             exit 1
         fi

         dd bs=1M if=/dev/mapper/$disk of=/root/dockerback

         fsck.ext4 -a /dev/mapper/$disk
             ret=$?
             if [ $ret -ne 0 -a $ret -ne 1 ]; then
                 exit 1
             fi

         fsck.ext4 -fn /dev/mapper/$disk
             ret=$?
             if [ $ret -ne 0 ]; then
                 exit 1
             fi
     done

     if [ $count -gt 5 ];then
         echo 3 > /proc/sys/vm/drop_caches
         sleep 1
         cat /proc/meminfo >> mem.txt
         echo "" >> mem.txt
         slabtop -o >> slab.txt
         echo "" >> slab.txt
         count=0
     fi
done

> 
> Thanks,
> 
> 						- Ted
> .
>