Re: {WHAT?} read checksum verification

[David Arendt <admin-/LHdS3kC8BfYtjvyW6yDsg@public.gmane.org>]

On 7/13/23 00:29, Peter Grandi wrote:
>> I used NILFS over ISCSI. I had random block corruption during
>> one week, silently destroying data until NILFS finally
>> crashed. First of all, I thought about a NILFS bug, so I
>> created a BTRFS volume
> I use both for main filesystem and backup for "diversity", and I
> value NILFS2 because it is very robust (I don't really use
> either filesystems snapshotting features).
So do I, therefor I said it was not NILFS fault.
>> and restored the backup from one week earlier to it. After
>> minutes, the BTRFS volume gave checksum errors, so the
>> culrprit was found, the ISCSI server.
> There used to be a good argument that checksumming (or
> compressing) data should be end-to-end and checksumming (or
> compressing) in the filesystem is a bit too much, but when LOGFS
> and NILFS/nILFS2 were designed I guess CPUs were too slow to
> checksum everything. Even excellent recent filesystems like F2FS
> don't do data integrity checking for various reasons though.
>
> In theory your iSCSI or its host-adapter should have told you
> about errors... Many can enable after-write verification (even
> if its quite expensive). Alternatively some people run regularly
> silent-corruption detecting daemons if their hardware does not
> report corruption or it escapes the relevant checks for various
> reasons:

The host adapter can return errors if underlying the disk itself returns 
them. If bits randomly flip on disk after being written, the host 
adapter can't know (at least not in non raid scenarios).

> https://indico.desy.de/event/257/contributions/58082/attachments/37574/46878/kelemen-2007-HEPiX-Silent_Corruptions.pdf
> https://storagemojo.com/2007/09/19/cerns-data-corruption-research/
>
>> [...] NILFS creates checksums on block writes. It would really
>> be a good addition to verify these checksums on read [...]
> It would be interesting to have data integrity checking or
> compression in NILFS2, and log-structured filesystem makes that
> easier (Btrfs code is rather complex instead), but modifying
> mature and stable filesystems is a risky thing...
>
> My understanding is that these checksums are not quite suitable
> for data integrity checks but are designed for log-sequence
> recovery, a bit like journal checksums for journal-based
> filesystems.
>
> https://www.spinics.net/lists/linux-nilfs/msg01063.html
> "nilfs2 store checksums for all data. However, at least the
> current implementation does not verify it when reading.
> Actually, the main purpose of the checksums is recovery after
> unexpected reboot; it does not suit for per-file data
> verification because the checksums are given per ``log''."

It think exactly this would be interesting, if checksum per log already 
exist, it would be good to verify these on read. As already said, I am 
not expecting to know on which file corruption occurred, but it would be 
nice to know that something nasty is going on.