Does data checksumming remain for files with No_COW file attribute?

Does data checksumming remain for files with No_COW file attribute?

[Alexander Tomokhov]

There is an unanswered question at stackexchange:
http://unix.stackexchange.com/q/303130/41572

Assume that we don't invoke mount options like nodatacow and nodatasum - operate only at file attributes level, which are controlled with lsattr/chattr. So does setting No_COW on a file (chattr +C) imply disabling data checksumming on it? The same way as mount options behave (nodatacow implies nodatasum)? Moreover may it disable checksumming only for newly written extents and keep for reading existing ones?

Re: Does data checksumming remain for files with No_COW file attribute?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

On Sat, Sep 24, 2016 at 03:26:14PM +0300, Alexander Tomokhov wrote:
> There is an unanswered question at stackexchange:
> http://unix.stackexchange.com/q/303130/41572
> 
> Assume that we don't invoke mount options like nodatacow and nodatasum - operate only at file attributes level, which are controlled with lsattr/chattr. So does setting No_COW on a file (chattr +C) imply disabling data checksumming on it? The same way as mount options behave (nodatacow implies nodatasum)? Moreover may it disable checksumming only for newly written extents and keep for reading existing ones?

   Yes, it does imply disabling checksums.

   If you have extents in the file when +C is set, then the flag is
ignored (even for new extents, as far as I know): you can only
usefully set +C on a file with no extents.

   So, to use +C, you either need to touch the file first, then set +C
on it, then open the file and write to it, or you need to have +C set
on the parent directory.

   Hugo.

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Hugo Mills             | Comic Sans goes into a bar, and the barman says, "We
hugo@... carfax.org.uk | don't serve your type here."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Alexander Tomokhov]

Ok, so data checksumming does not remain for newly created empty files with No_COW attribute. I think it's an important trait of Btrfs behavior and should be added to wiki. So that users are informed that disabling CoW on a per-file basis also loses checksum correctness of such file.

24.09.2016, 15:37, "Hugo Mills" <hugo@carfax.org.uk>:
> On Sat, Sep 24, 2016 at 03:26:14PM +0300, Alexander Tomokhov wrote:
>>  There is an unanswered question at stackexchange:
>>  http://unix.stackexchange.com/q/303130/41572
>>
>>  Assume that we don't invoke mount options like nodatacow and nodatasum - operate only at file attributes level, which are controlled with lsattr/chattr. So does setting No_COW on a file (chattr +C) imply disabling data checksumming on it? The same way as mount options behave (nodatacow implies nodatasum)? Moreover may it disable checksumming only for newly written extents and keep for reading existing ones?
>
>    Yes, it does imply disabling checksums.
>
>    If you have extents in the file when +C is set, then the flag is
> ignored (even for new extents, as far as I know): you can only
> usefully set +C on a file with no extents.
>
>    So, to use +C, you either need to touch the file first, then set +C
> on it, then open the file and write to it, or you need to have +C set
> on the parent directory.
>
>    Hugo.
>
>>  --
>>  To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>  the body of a message to majordomo@vger.kernel.org
>>  More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> Hugo Mills | Comic Sans goes into a bar, and the barman says, "We
> hugo@... carfax.org.uk | don't serve your type here."
> http://carfax.org.uk/ |
> PGP: E2AB1DE4 |

Re: Does data checksumming remain for files with No_COW file attribute?

[Adam Borowski]

On Sun, Sep 25, 2016 at 02:25:32AM +0300, Alexander Tomokhov wrote:
> Ok, so data checksumming does not remain for newly created empty files
> with No_COW attribute.  I think it's an important trait of Btrfs behavior
> and should be added to wiki.  So that users are informed that disabling
> CoW on a per-file basis also loses checksum correctness of such file.

Actually, it disables pretty much all btrfs features except for... CoW.

You lose:
* checksums
* compression
* safety against power loss (torn writes, etc)
* transactions (not that anyone uses them...)
* etc
But, CoW is still there.  Try it: make a subvolume, create a FS_NO_COW file
(preferably one big enough), snapshot the subvolume, filefrag -v both
copies.  Write to one of them, changing only a part of file.  Wait for
writeout, filefrag -v them again.

-- 
An imaginary friend squared is a real enemy.

Re: Does data checksumming remain for files with No_COW file attribute?

[Duncan]

Adam Borowski posted on Sun, 25 Sep 2016 01:50:14 +0200 as excerpted:

> On Sun, Sep 25, 2016 at 02:25:32AM +0300, Alexander Tomokhov wrote:
>> Ok, so data checksumming does not remain for newly created empty files
>> with No_COW attribute.  I think it's an important trait of Btrfs
>> behavior and should be added to wiki.  So that users are informed that
>> disabling CoW on a per-file basis also loses checksum correctness of
>> such file.
> 
> Actually, it disables pretty much all btrfs features except for... CoW.
> 
> You lose:
> * checksums
> * compression
> * safety against power loss (torn writes, etc)
> * transactions (not that anyone uses them...)
> * etc

> But, CoW is still there.

> Try it: make a subvolume, create a
> FS_NO_COW file (preferably one big enough), snapshot the subvolume,
> filefrag -v both copies.  Write to one of them, changing only a part of
> file.  Wait for writeout, filefrag -v them again.

That's because snapshots depend on COW.  If you don't snapshot the file 
(or otherwise create additional reflinks to it, using cp --reflink=always, 
for instance), it'll be NOCOW.  But because snapshots (and other forms of 
multiple reflink) depend on COW, taking a snapshot (or otherwise multi-
reflinking) and then writing to one copy forces what has been referred to 
on this list as COW1, a single COW to break the multi-reflink.

However, COW1 doesn't change the NOCOW attribute, and further writes to 
the same block of the NOCOW file will overwrite the now-current block, 
instead of COWING it... until the next snapshot (or another multi-reflink 
operation) locks it too in place, of course, after which another COW1 
will be required.

Which means otherwise NOCOW files that are both repeatedly overwritten 
and repeatedly snapshotted, with both happening at about the same rate or 
snapshots happening more frequently than rewrites, will tend to fragment 
almost as fast as if they hadn't been set NOCOW in the first place.

So NOCOW still has an effect -- as long as rewrites are coming in more 
frequently than snapshots.  However, if the file is repeatedly snapshotted 
at the same or faster rate than it is rewritten, all those COW1s due to 
the repeated snapshotting will pretty effectively nullify the NOCOW 
setting, even if it otherwise remains valid.

The other alternative, of course, is to avoid snapshotting your NOCOW 
files (which of course means losing send/receive, since send requires a 
read-only snapshot).  You can choose one or the other, but can't have 
both without one, NOCOW, yielding to the other.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Does data checksumming remain for files with No_COW file attribute?

[Adam Borowski]

On Sun, Sep 25, 2016 at 05:44:30AM +0000, Duncan wrote:
> Adam Borowski posted on Sun, 25 Sep 2016 01:50:14 +0200 as excerpted:
> > Actually, it disables pretty much all btrfs features except for... CoW.
> > 
> > You lose:
> > * checksums
> > * compression
> > * safety against power loss (torn writes, etc)
> > * transactions (not that anyone uses them...)
> > * etc
> 
> > But, CoW is still there.
> 
> That's because snapshots depend on COW.  If you don't snapshot the file 
> (or otherwise create additional reflinks to it, using cp --reflink=always, 
> for instance), it'll be NOCOW.  But because snapshots (and other forms of 
> multiple reflink) depend on COW, taking a snapshot (or otherwise multi-
> reflinking) and then writing to one copy forces what has been referred to 
> on this list as COW1, a single COW to break the multi-reflink.
> 
> However, COW1 doesn't change the NOCOW attribute, and further writes to 
> the same block of the NOCOW file will overwrite the now-current block, 
> instead of COWING it... until the next snapshot (or another multi-reflink 
> operation) locks it too in place, of course, after which another COW1 
> will be required.

There is no CoW unless you have more than one ref to the extent, as there is
no copying involved (other than when you write less than a whole block[1],
but that's the case for any filesystem).  Depending on FS_NOCOW_FL, you
either overwrite in place or write elsewhere then switch refs, but at no
point there is more than one live version.  You could call the latter "move
on write", perhaps.  On the other hand, with >1 reference, there's always
CoW.

[1]. The exact definition of "block" here varies.
-- 
An imaginary friend squared is a real enemy.

Re: Does data checksumming remain for files with No_COW file attribute?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Sat, 24 Sep 2016 15:26:14 +0300
Alexander Tomokhov <alexoundos@ya.ru> wrote:

> does setting No_COW on a file (chattr +C) imply disabling data checksumming on it?

Yes. IIRC the reasoning was that it's more difficult to track checksums of
data which is being overwritten in-place (as opposed to CoW).

> may it disable checksumming only for newly written extents and keep for reading existing ones?

You can't apply chattr +C to any files of non-zero length, so by definition
there won't be any pre-existing checksummed extents in that file.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Sat, Sep 24, 2016 at 05:40:12PM +0500, Roman Mamedov wrote:
> On Sat, 24 Sep 2016 15:26:14 +0300
> Alexander Tomokhov <alexoundos@ya.ru> wrote:
> 
> > does setting No_COW on a file (chattr +C) imply disabling data checksumming on it?
> 
> Yes. IIRC the reasoning was that it's more difficult to track checksums of
> data which is being overwritten in-place (as opposed to CoW).

   It's because you can't update the data and the checksum atomically
-- at some point in the writing process, they must be inconsistent.
This is considered a Bad Thing.

   Hugo.

> > may it disable checksumming only for newly written extents and keep for reading existing ones?
> 
> You can't apply chattr +C to any files of non-zero length, so by definition
> there won't be any pre-existing checksummed extents in that file.
> 
> -- 
> With respect,
> Roman



-- 
Hugo Mills             | Comic Sans goes into a bar, and the barman says, "We
hugo@... carfax.org.uk | don't serve your type here."
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

On Sat, 2016-09-24 at 12:43 +0000, Hugo Mills wrote:
>    It's because you can't update the data and the checksum atomically
> -- at some point in the writing process, they must be inconsistent.
> This is considered a Bad Thing.

It's not worse at all than simply not cheksuming... in both cases you
have no guarantee whether the data is bogus or not, but in with csums,
you can at least tell - only at the little cost, of false positives
when the data was fully and correctly written, but the csums not.


Cheers.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Goffredo Baroncelli]

On 09/24/2016 08:11 PM, Christoph Anton Mitterer wrote:
> On Sat, 2016-09-24 at 12:43 +0000, Hugo Mills wrote:
>>    It's because you can't update the data and the checksum atomically
>> -- at some point in the writing process, they must be inconsistent.
>> This is considered a Bad Thing.
> 
> It's not worse at all than simply not cheksuming... in both cases you
> have no guarantee whether the data is bogus or not, but in with csums,
> you can at least tell - only at the little cost, of false positives
> when the data was fully and correctly written, but the csums not.
> 

I think that the bigger cost is the lower performance due to the write of checksums.
> 
> Cheers.
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Does data checksumming remain for files with No_COW file attribute?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

On Sun, 2016-09-25 at 15:49 +0200, Goffredo Baroncelli wrote:
> I think that the bigger cost is the lower performance due to the
> write of checksums.

Which would make btrfs default mode of CoW+checksumming also
unusable...

If for anyone checksumming comes at a too high cost, he can simply
use nodatasum.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Austin S. Hemmelgarn]

On 2016-09-24 14:11, Christoph Anton Mitterer wrote:
> On Sat, 2016-09-24 at 12:43 +0000, Hugo Mills wrote:
>>    It's because you can't update the data and the checksum atomically
>> -- at some point in the writing process, they must be inconsistent.
>> This is considered a Bad Thing.
>
> It's not worse at all than simply not cheksuming... in both cases you
> have no guarantee whether the data is bogus or not, but in with csums,
> you can at least tell - only at the little cost, of false positives
> when the data was fully and correctly written, but the csums not.
Except that without checksums, you don't lose all the data in that block 
when torn write happens, with checksums without COW, you do.  Also, 
IIUC, due to how BTRFS implements checksums, you have a bad block for 
the entire duration of the write, because the stored checksum is for the 
old block, not the current one.  Whether or not that matters is of 
course usage dependent, but it's still worth noting.  Personally, I'd 
rather be able to read the file and see that it's corrupted instead of 
getting -EIO trying to do so (I also would rather we had something other 
than -EIO to return when checksum verification fails, but that's a 
separate issue).

Re: Does data checksumming remain for files with No_COW file attribute?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

On Sat, 2016-09-24 at 17:40 +0500, Roman Mamedov wrote:
> Yes. IIRC the reasoning was that it's more difficult to track
> checksums of
> data which is being overwritten in-place (as opposed to CoW).
AFAIU it wouldn't be more difficult, since the meta-data itself is
still subject to CoW...

There's just no guarantee in the case of a crash, that checksum and
data match (which is IMO however a small price to pay - especially as
the data is in that case and without CoW anyway not guaranteed to be
valid - compared to all sorts of other silent corruptions against which
checksums protect... not to talk about the ability to actually repair
files in case of RAID inconsistencies.


> You can't apply chattr +C to any files of non-zero length, so by
> definition
> there won't be any pre-existing checksummed extents in that file.

Speaking of which,... can't one modify chattr properly so that it gives
an error message and $? != 0 in that case?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: Does data checksumming remain for files with No_COW file attribute?

[Adam Borowski]

On Sat, Sep 24, 2016 at 08:09:26PM +0200, Christoph Anton Mitterer wrote:
> On Sat, 2016-09-24 at 17:40 +0500, Roman Mamedov wrote:
> > Yes. IIRC the reasoning was that it's more difficult to track checksums
> > of data which is being overwritten in-place (as opposed to CoW).
> AFAIU it wouldn't be more difficult, since the meta-data itself is
> still subject to CoW...
> 
> There's just no guarantee in the case of a crash, that checksum and
> data match (which is IMO however a small price to pay - especially as
> the data is in that case and without CoW anyway not guaranteed to be
> valid - compared to all sorts of other silent corruptions against which
> checksums protect... not to talk about the ability to actually repair
> files in case of RAID inconsistencies.

This would require teaching btrfs that, in some cases, a csum mismatch is no
big thing and it can legitimately return junk data (like most other
filesystems) rather than complaining.  Same for scrub and btrfs check.
For now, there's a nice simple rule: bad csum is bad.

> > You can't apply chattr +C to any files of non-zero length, so by
> > definition there won't be any pre-existing checksummed extents in that
> > file.
> 
> Speaking of which,... can't one modify chattr properly so that it gives
> an error message and $? != 0 in that case?

chattr has no knowledge of btrfs whatsoever, and has no filesystem-specific
code.  It comes from e2fsprogs and from its internal naming of flags, I see
it believes everything to be ext* -- but it doesn't actually have anything
specific to ext*, either.

Thus, any other filesystem can make use of +C (FS_NOCOW_FL), and there's no
reason the "+C doesn't work on non-empty files" rule needs to hold on btrfs
in the future.


Meow!
-- 
An imaginary friend squared is a real enemy.

Re: Does data checksumming remain for files with No_COW file attribute?

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

On Sat, 2016-09-24 at 23:44 +0200, Adam Borowski wrote:
> This would require teaching btrfs that, in some cases, a csum
> mismatch is no
> big thing and it can legitimately return junk data (like most other
> filesystems) rather than complaining.  Same for scrub and btrfs
> check.
Well, I see no point in returning data that is not "guaranteed" to be
valid.
In case of a crash, one must assume that the data is bogus, and that's
what the program should get.

> chattr has no knowledge of btrfs whatsoever, and has no filesystem-
> specific
> code.  It comes from e2fsprogs and from its internal naming of flags,
> I see
> it believes everything to be ext* -- but it doesn't actually have
> anything
> specific to ext*, either.
> 
> Thus, any other filesystem can make use of +C (FS_NOCOW_FL), and
> there's no
> reason the "+C doesn't work on non-empty files" rule needs to hold on
> btrfs
> in the future.

Well chattr must probably call some btrfs ioctl or so... and that
should simply give an error back... and chattr at least print some
generic error message.
Or this should be rather replaced with a btrfs specific tool.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]