What is status of metadata_csum

What is status of metadata_csum

[Oleksij Rempel]

What is metadata_csum status?

i see there are benchmark updates on wiki page, but no updates for
e2fsprogs. 1.43-WIP seems to be really old.
-- 
Regards,
Oleksij

Re: What is status of metadata_csum

[Darrick J. Wong]

On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
> What is metadata_csum status?
> 
> i see there are benchmark updates on wiki page, but no updates for
> e2fsprogs. 1.43-WIP seems to be really old.

Still waiting for 1.43.  I don't know if there are early adopters running
1.43-WIP without complaint, or if simply nobody's using it at all. :/

--D
> -- 
> Regards,
> Oleksij
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: What is status of metadata_csum

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Am 17.01.2014 03:00, schrieb Darrick J. Wong:
> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
>> What is metadata_csum status?
>>
>> i see there are benchmark updates on wiki page, but no updates for
>> e2fsprogs. 1.43-WIP seems to be really old.
> 
> Still waiting for 1.43.  I don't know if there are early adopters running
> 1.43-WIP without complaint, or if simply nobody's using it at all. :/

I didn't tested current master.git - my bad. It is not outdated :)

Beside, the wiki should be corrected. If you enable 64bit, it will need
extents too.

Are there any existing tools for error injection for file systems?
Random read error from libata or silent data corruptions?
-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 295 bytes --]

Re: What is status of metadata_csum

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 2170 bytes --]

Am 17.01.2014 09:18, schrieb Oleksij Rempel:
> Am 17.01.2014 03:00, schrieb Darrick J. Wong:
>> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
>>> What is metadata_csum status?
>>>
>>> i see there are benchmark updates on wiki page, but no updates for
>>> e2fsprogs. 1.43-WIP seems to be really old.
>>
>> Still waiting for 1.43.  I don't know if there are early adopters running
>> 1.43-WIP without complaint, or if simply nobody's using it at all. :/
> 
> I didn't tested current master.git - my bad. It is not outdated :)
> 
> Beside, the wiki should be corrected. If you enable 64bit, it will need
> extents too.
> 
> Are there any existing tools for error injection for file systems?
> Random read error from libata or silent data corruptions?
> 

I was playing with metadata_csum trying to kill my system and now i have
some questions:

first the test:
#find block number
sudo ./debugfs/debugfs -R "imap <3991825>" /dev/sdf1

#copy block
sudo dd if=/dev/sdf1 bs=4K count=1 skip=15958163 of=corrupt_block.bin

#change one byte
ghex corrupt_block.bin

#kill inode
sudo dd of=/dev/sdf1 bs=4K count=1 seek=15958163 if=corrupt_block.bin

sudo ./debugfs/debugfs -R "stat <3991825>" /dev/sdf1
#stat: Inode checksum does not match inode while reading inode 3991825

so, inode is corrupt:
- ext4 can detect inode corruption. Great work! I like it :)
- in this case file will not be shown by Nautilus.
- ls will show some thing like this:

4130898 drwx------  8 oleksij oleksij      4096 Jan 17 09:31 Bilder
      ? -?????????  ? ?       ?               ?            ?
ich-einfach_unverbesserlich_2.mkv
3991826 -rw-rw-r--  1 oleksij oleksij 622038707 Jan 17 11:12 igor.mkv
3989506 drwxr-xr-x 26 oleksij oleksij      4096 Okt  9 23:11 linux

it means, i know there is some thing wrong, but i do not know which inode

- If fs is not fixed, file is lost? are blocks still allocated to this
inode?
- debufs will not allow me to edit inode corrupt inode.
- If i run fsck -y file is lost. Most user wont be able to recover it.
- how can i sync it with back up software?





-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 295 bytes --]

Re: What is status of metadata_csum

[Darrick J. Wong]

On Fri, Jan 17, 2014 at 11:39:07AM +0100, Oleksij Rempel wrote:
> Am 17.01.2014 09:18, schrieb Oleksij Rempel:
> > Am 17.01.2014 03:00, schrieb Darrick J. Wong:
> >> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
> >>> What is metadata_csum status?
> >>>
> >>> i see there are benchmark updates on wiki page, but no updates for
> >>> e2fsprogs. 1.43-WIP seems to be really old.
> >>
> >> Still waiting for 1.43.  I don't know if there are early adopters running
> >> 1.43-WIP without complaint, or if simply nobody's using it at all. :/
> > 
> > I didn't tested current master.git - my bad. It is not outdated :)
> > 
> > Beside, the wiki should be corrected. If you enable 64bit, it will need
> > extents too.
> > 
> > Are there any existing tools for error injection for file systems?
> > Random read error from libata or silent data corruptions?
> > 
> 
> I was playing with metadata_csum trying to kill my system and now i have
> some questions:
> 
> first the test:
> #find block number
> sudo ./debugfs/debugfs -R "imap <3991825>" /dev/sdf1
> 
> #copy block
> sudo dd if=/dev/sdf1 bs=4K count=1 skip=15958163 of=corrupt_block.bin
> 
> #change one byte
> ghex corrupt_block.bin
> 
> #kill inode
> sudo dd of=/dev/sdf1 bs=4K count=1 seek=15958163 if=corrupt_block.bin
> 
> sudo ./debugfs/debugfs -R "stat <3991825>" /dev/sdf1
> #stat: Inode checksum does not match inode while reading inode 3991825
> 
> so, inode is corrupt:
> - ext4 can detect inode corruption. Great work! I like it :)
> - in this case file will not be shown by Nautilus.
> - ls will show some thing like this:
> 
> 4130898 drwx------  8 oleksij oleksij      4096 Jan 17 09:31 Bilder
>       ? -?????????  ? ?       ?               ?            ?
> ich-einfach_unverbesserlich_2.mkv
> 3991826 -rw-rw-r--  1 oleksij oleksij 622038707 Jan 17 11:12 igor.mkv
> 3989506 drwxr-xr-x 26 oleksij oleksij      4096 Okt  9 23:11 linux
> 
> it means, i know there is some thing wrong, but i do not know which inode

Unfortunately, either the whole stat call succeeds or it doesn't.  Your best
bet to find the inode is:
debugfs -n -R 'stat /path/within/fs' /dev/sdXXX

> - If fs is not fixed, file is lost? are blocks still allocated to this
> inode?

Yes, the blocks are still allocated.

> - debufs will not allow me to edit inode corrupt inode.

debugfs -n (disables checksum verification)

> - If i run fsck -y file is lost. Most user wont be able to recover it.

If you answer 'n' to the first prompt about the checksum failure, e2fsck will
check the rest of the inode.  If no other problems are found, it'll ask to
correct the checksum.

I've wondered if perhaps e2fsck ought to do something like this when there's a
checksum problem:

1. If the user didn't prohibit clearing the inode as the first potential
   action, ask if e2fsck should just clear the inode.
2. If not, run the regular checks.
3. If there's still a checksum error, ask to fix the checksum.
4. If the user doesn't want to fix it, ask to clear the inode.

iow, e2fsck --verify-checksums-last /dev/sdXX

> - how can i sync it with back up software?

fsck -y to clear the bad inode, then restore from backup.

--D

Re: What is status of metadata_csum

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 3594 bytes --]

Am 17.01.2014 20:21, schrieb Darrick J. Wong:
> On Fri, Jan 17, 2014 at 11:39:07AM +0100, Oleksij Rempel wrote:
>> Am 17.01.2014 09:18, schrieb Oleksij Rempel:
>>> Am 17.01.2014 03:00, schrieb Darrick J. Wong:
>>>> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
>>>>> What is metadata_csum status?
>>>>>
>>>>> i see there are benchmark updates on wiki page, but no updates for
>>>>> e2fsprogs. 1.43-WIP seems to be really old.
>>>>
>>>> Still waiting for 1.43.  I don't know if there are early adopters running
>>>> 1.43-WIP without complaint, or if simply nobody's using it at all. :/
>>>
>>> I didn't tested current master.git - my bad. It is not outdated :)
>>>
>>> Beside, the wiki should be corrected. If you enable 64bit, it will need
>>> extents too.
>>>
>>> Are there any existing tools for error injection for file systems?
>>> Random read error from libata or silent data corruptions?
>>>
>>
>> I was playing with metadata_csum trying to kill my system and now i have
>> some questions:
>>
>> first the test:
>> #find block number
>> sudo ./debugfs/debugfs -R "imap <3991825>" /dev/sdf1
>>
>> #copy block
>> sudo dd if=/dev/sdf1 bs=4K count=1 skip=15958163 of=corrupt_block.bin
>>
>> #change one byte
>> ghex corrupt_block.bin
>>
>> #kill inode
>> sudo dd of=/dev/sdf1 bs=4K count=1 seek=15958163 if=corrupt_block.bin
>>
>> sudo ./debugfs/debugfs -R "stat <3991825>" /dev/sdf1
>> #stat: Inode checksum does not match inode while reading inode 3991825
>>
>> so, inode is corrupt:
>> - ext4 can detect inode corruption. Great work! I like it :)
>> - in this case file will not be shown by Nautilus.
>> - ls will show some thing like this:
>>
>> 4130898 drwx------  8 oleksij oleksij      4096 Jan 17 09:31 Bilder
>>       ? -?????????  ? ?       ?               ?            ?
>> ich-einfach_unverbesserlich_2.mkv
>> 3991826 -rw-rw-r--  1 oleksij oleksij 622038707 Jan 17 11:12 igor.mkv
>> 3989506 drwxr-xr-x 26 oleksij oleksij      4096 Okt  9 23:11 linux
>>
>> it means, i know there is some thing wrong, but i do not know which inode
> 
> Unfortunately, either the whole stat call succeeds or it doesn't.  Your best
> bet to find the inode is:
> debugfs -n -R 'stat /path/within/fs' /dev/sdXXX

Ok, looks like user space apps should handle errors in different whey. I
mean some thing like, show notification about probable FS errors and so on.

>> - If fs is not fixed, file is lost? are blocks still allocated to this
>> inode?
> 
> Yes, the blocks are still allocated.
> 
>> - debufs will not allow me to edit inode corrupt inode.
> 
> debugfs -n (disables checksum verification)
> 
>> - If i run fsck -y file is lost. Most user wont be able to recover it.
> 
> If you answer 'n' to the first prompt about the checksum failure, e2fsck will
> check the rest of the inode.  If no other problems are found, it'll ask to
> correct the checksum.
> 
> I've wondered if perhaps e2fsck ought to do something like this when there's a
> checksum problem:
> 
> 1. If the user didn't prohibit clearing the inode as the first potential
>    action, ask if e2fsck should just clear the inode.
> 2. If not, run the regular checks.
> 3. If there's still a checksum error, ask to fix the checksum.
> 4. If the user doesn't want to fix it, ask to clear the inode.

sounds good.

> iow, e2fsck --verify-checksums-last /dev/sdXX

this mean?

PS: More testing results from today. GRUB fails to install if
metadata_csum enabled. Should it be reported to GRUB devs?

-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 295 bytes --]

Re: What is status of metadata_csum

[Darrick J. Wong]

On Fri, Jan 17, 2014 at 09:13:11PM +0100, Oleksij Rempel wrote:
> Am 17.01.2014 20:21, schrieb Darrick J. Wong:
> > On Fri, Jan 17, 2014 at 11:39:07AM +0100, Oleksij Rempel wrote:
> >> Am 17.01.2014 09:18, schrieb Oleksij Rempel:
> >>> Am 17.01.2014 03:00, schrieb Darrick J. Wong:
> >>>> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
> >>>>> What is metadata_csum status?
> >>>>>
> >>>>> i see there are benchmark updates on wiki page, but no updates for
> >>>>> e2fsprogs. 1.43-WIP seems to be really old.
> >>>>
> >>>> Still waiting for 1.43.  I don't know if there are early adopters running
> >>>> 1.43-WIP without complaint, or if simply nobody's using it at all. :/
> >>>
> >>> I didn't tested current master.git - my bad. It is not outdated :)
> >>>
> >>> Beside, the wiki should be corrected. If you enable 64bit, it will need
> >>> extents too.
> >>>
> >>> Are there any existing tools for error injection for file systems?
> >>> Random read error from libata or silent data corruptions?
> >>>
> >>
> >> I was playing with metadata_csum trying to kill my system and now i have
> >> some questions:
> >>
> >> first the test:
> >> #find block number
> >> sudo ./debugfs/debugfs -R "imap <3991825>" /dev/sdf1
> >>
> >> #copy block
> >> sudo dd if=/dev/sdf1 bs=4K count=1 skip=15958163 of=corrupt_block.bin
> >>
> >> #change one byte
> >> ghex corrupt_block.bin
> >>
> >> #kill inode
> >> sudo dd of=/dev/sdf1 bs=4K count=1 seek=15958163 if=corrupt_block.bin
> >>
> >> sudo ./debugfs/debugfs -R "stat <3991825>" /dev/sdf1
> >> #stat: Inode checksum does not match inode while reading inode 3991825
> >>
> >> so, inode is corrupt:
> >> - ext4 can detect inode corruption. Great work! I like it :)
> >> - in this case file will not be shown by Nautilus.
> >> - ls will show some thing like this:
> >>
> >> 4130898 drwx------  8 oleksij oleksij      4096 Jan 17 09:31 Bilder
> >>       ? -?????????  ? ?       ?               ?            ?
> >> ich-einfach_unverbesserlich_2.mkv
> >> 3991826 -rw-rw-r--  1 oleksij oleksij 622038707 Jan 17 11:12 igor.mkv
> >> 3989506 drwxr-xr-x 26 oleksij oleksij      4096 Okt  9 23:11 linux
> >>
> >> it means, i know there is some thing wrong, but i do not know which inode
> > 
> > Unfortunately, either the whole stat call succeeds or it doesn't.  Your best
> > bet to find the inode is:
> > debugfs -n -R 'stat /path/within/fs' /dev/sdXXX
> 
> Ok, looks like user space apps should handle errors in different whey. I
> mean some thing like, show notification about probable FS errors and so on.

AIX has ECORRUPT; maybe we should define that for the kernel/glibc?

> >> - If fs is not fixed, file is lost? are blocks still allocated to this
> >> inode?
> > 
> > Yes, the blocks are still allocated.
> > 
> >> - debufs will not allow me to edit inode corrupt inode.
> > 
> > debugfs -n (disables checksum verification)
> > 
> >> - If i run fsck -y file is lost. Most user wont be able to recover it.
> > 
> > If you answer 'n' to the first prompt about the checksum failure, e2fsck will
> > check the rest of the inode.  If no other problems are found, it'll ask to
> > correct the checksum.
> > 
> > I've wondered if perhaps e2fsck ought to do something like this when there's a
> > checksum problem:
> > 
> > 1. If the user didn't prohibit clearing the inode as the first potential
> >    action, ask if e2fsck should just clear the inode.
> > 2. If not, run the regular checks.
> > 3. If there's still a checksum error, ask to fix the checksum.
> > 4. If the user doesn't want to fix it, ask to clear the inode.
> 
> sounds good.
> 
> > iow, e2fsck --verify-checksums-last /dev/sdXX
> 
> this mean?

It would be a new flag to change the fsck behavior.

> PS: More testing results from today. GRUB fails to install if
> metadata_csum enabled. Should it be reported to GRUB devs?

Yes, please report the bug to them.  metadata_csum is a ROCOMPAT feature, which
means that grub will need to learn about checksums if it wants to do anything
other than rewrites of existing file blocks.

--D

Re: What is status of metadata_csum

[Oleksij Rempel]

[-- Attachment #1: Type: text/plain, Size: 2928 bytes --]

Am 17.01.2014 21:58, schrieb Darrick J. Wong:
> On Fri, Jan 17, 2014 at 09:13:11PM +0100, Oleksij Rempel wrote:
>> Am 17.01.2014 20:21, schrieb Darrick J. Wong:
>>> On Fri, Jan 17, 2014 at 11:39:07AM +0100, Oleksij Rempel wrote:
>>>> Am 17.01.2014 09:18, schrieb Oleksij Rempel:
>>>>> Am 17.01.2014 03:00, schrieb Darrick J. Wong:
>>>>>> On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
>>>>>>> What is metadata_csum status?
>>>>>>>
>>>>>>> i see there are benchmark updates on wiki page, but no updates for
>>>>>>> e2fsprogs. 1.43-WIP seems to be really old.
>>>>>>
>>>>>> Still waiting for 1.43.  I don't know if there are early adopters running
>>>>>> 1.43-WIP without complaint, or if simply nobody's using it at all. :/
>>>>>
>>>>> I didn't tested current master.git - my bad. It is not outdated :)
>>>>>
>>>>> Beside, the wiki should be corrected. If you enable 64bit, it will need
>>>>> extents too.
>>>>>
>>>>> Are there any existing tools for error injection for file systems?
>>>>> Random read error from libata or silent data corruptions?
>>>>>
>>>>
>>>> I was playing with metadata_csum trying to kill my system and now i have
>>>> some questions:
>>>>
>>>> first the test:
>>>> #find block number
>>>> sudo ./debugfs/debugfs -R "imap <3991825>" /dev/sdf1
>>>>
>>>> #copy block
>>>> sudo dd if=/dev/sdf1 bs=4K count=1 skip=15958163 of=corrupt_block.bin
>>>>
>>>> #change one byte
>>>> ghex corrupt_block.bin
>>>>
>>>> #kill inode
>>>> sudo dd of=/dev/sdf1 bs=4K count=1 seek=15958163 if=corrupt_block.bin
>>>>
>>>> sudo ./debugfs/debugfs -R "stat <3991825>" /dev/sdf1
>>>> #stat: Inode checksum does not match inode while reading inode 3991825
>>>>
>>>> so, inode is corrupt:
>>>> - ext4 can detect inode corruption. Great work! I like it :)
>>>> - in this case file will not be shown by Nautilus.
>>>> - ls will show some thing like this:
>>>>
>>>> 4130898 drwx------  8 oleksij oleksij      4096 Jan 17 09:31 Bilder
>>>>       ? -?????????  ? ?       ?               ?            ?
>>>> ich-einfach_unverbesserlich_2.mkv
>>>> 3991826 -rw-rw-r--  1 oleksij oleksij 622038707 Jan 17 11:12 igor.mkv
>>>> 3989506 drwxr-xr-x 26 oleksij oleksij      4096 Okt  9 23:11 linux
>>>>
>>>> it means, i know there is some thing wrong, but i do not know which inode
>>>
>>> Unfortunately, either the whole stat call succeeds or it doesn't.  Your best
>>> bet to find the inode is:
>>> debugfs -n -R 'stat /path/within/fs' /dev/sdXXX
>>
>> Ok, looks like user space apps should handle errors in different whey. I
>> mean some thing like, show notification about probable FS errors and so on.
> 
> AIX has ECORRUPT; maybe we should define that for the kernel/glibc?

It would be awesome! :)

And some return code on mount if FS is not clean. Fsck is done
automatically on interal drives, but never on hotplug devices.

-- 
Regards,
Oleksij


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 295 bytes --]

Re: What is status of metadata_csum

[Darrick J. Wong]

On Fri, Jan 17, 2014 at 09:18:29AM +0100, Oleksij Rempel wrote:
> Am 17.01.2014 03:00, schrieb Darrick J. Wong:
> > On Thu, Jan 16, 2014 at 11:57:20AM +0100, Oleksij Rempel wrote:
> >> What is metadata_csum status?
> >>
> >> i see there are benchmark updates on wiki page, but no updates for
> >> e2fsprogs. 1.43-WIP seems to be really old.
> > 
> > Still waiting for 1.43.  I don't know if there are early adopters running
> > 1.43-WIP without complaint, or if simply nobody's using it at all. :/
> 
> I didn't tested current master.git - my bad. It is not outdated :)
> 
> Beside, the wiki should be corrected. If you enable 64bit, it will need
> extents too.

I updated the howto.  mke2fs (as of 1.42.9) should no longer allow
extents,64bit on a filesystem.

--D
> 
> Are there any existing tools for error injection for file systems?
> Random read error from libata or silent data corruptions?
> -- 
> Regards,
> Oleksij
>