Ideas for a feature implementation

Ideas for a feature implementation

[Vimal A R]

Hello,

I came across the to-do list at https://btrfs.wiki.kernel.org/index.php/Project_ideas and would like to know if this list is updated and recent.

I am looking for a project idea for my under graduate degree which can be completed in around 3-4 months. Are there any suggestions and ideas to help me further?

Thank you,
Vimal

Re: Ideas for a feature implementation

[Austin S Hemmelgarn]

On 08/10/2014 03:21 PM, Vimal A R wrote:
> Hello,
> 
> I came across the to-do list at https://btrfs.wiki.kernel.org/index.php/Project_ideas and would like to know if this list is updated and recent.
> 
> I am looking for a project idea for my under graduate degree which can be completed in around 3-4 months. Are there any suggestions and ideas to help me further?
> 
> Thank you,
> Vimal
It's not really listed there (though some of the projects there might be
considered subsets of it), but improved parallelization for the
multi-device setups is one thing that I know that a lot of people would
like to see.

Another thing that isn't listed there, that I would personally love to
see is support for secure file deletion.  To be truly secure though,
this would need to hook into the COW logic so that files marked for
secure deletion can't be reflinked (maybe make the automatically NOCOW
instead, and don't allow snapshots?), and when they get written to, the
blocks that get COW'ed have the old block overwritten.

Re: Ideas for a feature implementation

[Chris Murphy]

On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> 
> Another thing that isn't listed there, that I would personally love to
> see is support for secure file deletion.  To be truly secure though,
> this would need to hook into the COW logic so that files marked for
> secure deletion can't be reflinked (maybe make the automatically NOCOW
> instead, and don't allow snapshots?), and when they get written to, the
> blocks that get COW'ed have the old block overwritten.

If the file is reflinked or snapshot, then it can it be secure deleted? Because what does it mean to secure delete a file when there's a completely independent file pointing to the same physical blocks? What if someone else owns that independent file? Does the reflink copy get rm'd as well? Or does the file remain, but its blocks are zero'd/corrupted?

For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open question when the data is actually irretrievable. It may be seconds, but could be much longer (hours?) so I'm not sure if it's useful. On HDD's using SMR it's not necessarily a given an overwrite will work there either.


Chris Murphy

Re: Ideas for a feature implementation

[Austin S Hemmelgarn]

On 08/11/2014 04:27 PM, Chris Murphy wrote:
> 
> On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
> 
>> 
>> Another thing that isn't listed there, that I would personally
>> love to see is support for secure file deletion.  To be truly
>> secure though, this would need to hook into the COW logic so that
>> files marked for secure deletion can't be reflinked (maybe make
>> the automatically NOCOW instead, and don't allow snapshots?), and
>> when they get written to, the blocks that get COW'ed have the old
>> block overwritten.
> 
> If the file is reflinked or snapshot, then it can it be secure
> deleted? Because what does it mean to secure delete a file when
> there's a completely independent file pointing to the same physical
> blocks? What if someone else owns that independent file? Does the
> reflink copy get rm'd as well? Or does the file remain, but its
> blocks are zero'd/corrupted?
The semantics that I would expect would be that the extents can't be
reflinked, and when snapshotted the whole file gets COW'ed, and then
inherits the secure deletion flag, possibly with another flag saying
that the user can't disable the secure deletion flag.
> 
> For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open
> question when the data is actually irretrievable. It may be
> seconds, but could be much longer (hours?) so I'm not sure if it's
> useful. On HDD's using SMR it's not necessarily a given an
> overwrite will work there either.
By secure deletion, I don't mean make the data absolutely
unrecoverable by any means, I mean make it functionally impractical
for someone without low-level access to and/or extensive knowledge of
the hardware to recover the data; that is, more secure than simply
unlinking the file, but obviously less than (for example) the
application of thermite to the disk platters.  I'm talking the rough
equivalent of wiping the data from RAM.

Anyone who is truly security minded should be using whole disk
encryption anyway, but even then you have the data accessible from the
running OS.

Re: Ideas for a feature implementation

[Chris Murphy]

On Aug 11, 2014, at 8:27 PM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:

> On 08/11/2014 04:27 PM, Chris Murphy wrote:
>> 
>> On Aug 10, 2014, at 8:53 PM, Austin S Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
>> 
>>> 
>>> Another thing that isn't listed there, that I would personally
>>> love to see is support for secure file deletion.  To be truly
>>> secure though, this would need to hook into the COW logic so that
>>> files marked for secure deletion can't be reflinked (maybe make
>>> the automatically NOCOW instead, and don't allow snapshots?), and
>>> when they get written to, the blocks that get COW'ed have the old
>>> block overwritten.
>> 
>> If the file is reflinked or snapshot, then it can it be secure
>> deleted? Because what does it mean to secure delete a file when
>> there's a completely independent file pointing to the same physical
>> blocks? What if someone else owns that independent file? Does the
>> reflink copy get rm'd as well? Or does the file remain, but its
>> blocks are zero'd/corrupted?
> The semantics that I would expect would be that the extents can't be
> reflinked, and when snapshotted the whole file gets COW'ed, and then
> inherits the secure deletion flag, possibly with another flag saying
> that the user can't disable the secure deletion flag.

Ahh OK I was thinking of a secure delete command (or an option to rm that indicates secure delete). You're suggesting one or more flags that makes for secure file handling, not just delete, affecting: a.) copied b.) moved, c.) snapshot/reflinked, d.) deleted.  So if deleted, a regular rm would "see" the xattr and do a secure delete; and the xattr would inhibit or limit the others.

While a reflink or normal copy could be inhibited, the snapshot case seems more difficult because it just creates a new tree. It's not scanning the tree for files/folders with xattr, which would have to be done to go retroactively remove the file set with the secure delete flag - could be really slow. And what if the snapshot is made read-only?

Strictly secure delete, e.g. rm -s, would be more straightforward than a flag affecting other filesystem operations.

>> For SSDs, whether it's an overwrite or an FITRIM ioctl it's an open
>> question when the data is actually irretrievable. It may be
>> seconds, but could be much longer (hours?) so I'm not sure if it's
>> useful. On HDD's using SMR it's not necessarily a given an
>> overwrite will work there either.
> By secure deletion, I don't mean make the data absolutely
> unrecoverable by any means, I mean make it functionally impractical
> for someone without low-level access to and/or extensive knowledge of
> the hardware to recover the data; that is, more secure than simply
> unlinking the file, but obviously less than (for example) the
> application of thermite to the disk platters.  I'm talking the rough
> equivalent of wiping the data from RAM.
> 
> Anyone who is truly security minded should be using whole disk
> encryption anyway, but even then you have the data accessible from the
> running OS.

Seems straightforward for any file system already supporting discard. This even has a useful application for thinly provisioned storage and large files where you'd want the underlying logical layer to free up extents sooner than later - even if you didn't care about the security aspect.

But for that matter, on SSDs right now you can rm the file and then fstrim the file system to get the same effect.

Chris Murphy

Re: Ideas for a feature implementation

[David Pottage]

On 11/08/14 03:53, Austin S Hemmelgarn wrote:

> Another thing that isn't listed there, that I would personally love to
> see is support for secure file deletion.  To be truly secure though,
> this would need to hook into the COW logic so that files marked for
> secure deletion can't be reflinked (maybe make the automatically NOCOW
> instead, and don't allow snapshots?), and when they get written to, the
> blocks that get COW'ed have the old block overwritten.
How would secure deletion interact with file de-duplication?

For example suppose you and I are both users on a multi user system. We 
both obtain copies of the same file independently, and save that file to 
our home directories.

A background process notices that both files are the same and 
de-duplicates them. This means that both your file and mine point to the 
same blocks on disc. This is exactly the same as would happen if you 
made a COW copy of your file, transferred ownership to me, and I moved 
it into my home dir.

You then decide to secure delete your copy of the file. What happens to 
mine? If it gets removed, then you have just deleted a file you don't 
own, if it does not then the file-system has broken the contract to 
secure delete a file when you asked it to.

Also, what happens if the two files have similar portions, but they are 
not identical. For example, if you download and ISO image for ubuntu, 
and I download the ISO for kubuntu (at the same version). There will be 
a lot of sections that are the same, because they will contain a lot of 
packages in common, so there will be large gains in de-duplicating the 
similar parts, but most people would consider the files to be different.

Could this mean that if you secure delete your ubuntu iso, then portions 
of my kubuntu iso might become corrupt?

Even if we limit secure delete to root, then we still leave the risk of 
unintentonaly breaking user files, because non-one realised that all or 
part of the file appears in other files via de-duplication. In any case 
if secure delete is limited to root, then most people would not find it 
useful. (or they would use sudo to do it, which brings us back to the 
same problems).

Basically, I think that file secure deletion as a concept is not 
compatible with a 5th generation file system. If you relay want to 
securely remove a file, then copy the stuff you need elsewhere, and put 
the disc in the crusher. Alternatively put the filesystem in an encypted 
container, and then reformat the disc with a different encryption key.

-- 
David Pottage.

Re: Ideas for a feature implementation

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3749 bytes --]

On 2014-08-12 11:52, David Pottage wrote:
> 
> On 11/08/14 03:53, Austin S Hemmelgarn wrote:
> 
>> Another thing that isn't listed there, that I would personally love to
>> see is support for secure file deletion.  To be truly secure though,
>> this would need to hook into the COW logic so that files marked for
>> secure deletion can't be reflinked (maybe make the automatically NOCOW
>> instead, and don't allow snapshots?), and when they get written to, the
>> blocks that get COW'ed have the old block overwritten.
> How would secure deletion interact with file de-duplication?
> 
> For example suppose you and I are both users on a multi user system. We
> both obtain copies of the same file independently, and save that file to
> our home directories.
> 
> A background process notices that both files are the same and
> de-duplicates them. This means that both your file and mine point to the
> same blocks on disc. This is exactly the same as would happen if you
> made a COW copy of your file, transferred ownership to me, and I moved
> it into my home dir.
> 
> You then decide to secure delete your copy of the file. What happens to
> mine? If it gets removed, then you have just deleted a file you don't
> own, if it does not then the file-system has broken the contract to
> secure delete a file when you asked it to.
> 
> Also, what happens if the two files have similar portions, but they are
> not identical. For example, if you download and ISO image for ubuntu,
> and I download the ISO for kubuntu (at the same version). There will be
> a lot of sections that are the same, because they will contain a lot of
> packages in common, so there will be large gains in de-duplicating the
> similar parts, but most people would consider the files to be different.
> 
> Could this mean that if you secure delete your ubuntu iso, then portions
> of my kubuntu iso might become corrupt?
> 
You could work around this by marking the extent, instead of the file
(marking a file would mark all of it's extents), and then checking for
that marking when the extent is freed (ie, nobody refers to it anymore).
While this approach might not seem useful to most people, there are
practical use cases for it (even without whole disk encryption).
It would be pretty easy actually to integrate this globally for a
file-system as a mount option.
> Even if we limit secure delete to root, then we still leave the risk of
> unintentonaly breaking user files, because non-one realised that all or
> part of the file appears in other files via de-duplication. In any case
> if secure delete is limited to root, then most people would not find it
> useful. (or they would use sudo to do it, which brings us back to the
> same problems).
> 
> Basically, I think that file secure deletion as a concept is not
> compatible with a 5th generation file system. If you relay want to
> securely remove a file, then copy the stuff you need elsewhere, and put
> the disc in the crusher. Alternatively put the filesystem in an encypted
> container, and then reformat the disc with a different encryption key.
> 
While I agree that the traditional notion of secure deletion doesn't fit
in the current generation of file systems, there is still a need for COW
filesystems to be able to prevent sensitive data from being exposed
during run-time.  On any current BTRFS filesystem, it is still possible
to find blocks that have been COW'ed (assuming discard is turned off)
and have no referents, possibly long after the block itself is freed,
and especially if the volume is much larger than the stored data set
(like a large majority of desktop users these days) or the workload is
not write intensive.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]

Re: Ideas for a feature implementation

[Brendan Hide]

On 2014/08/12 17:52, David Pottage wrote:
>
> [snip] ... if it does not then the file-system has broken the contract 
> to secure delete a file when you asked it to.
This is a technicality - and it has not necessarily "broken the contract".

I think the correct thing to do would be to securely delete the metadata 
referring to that file. That would satisfy the concept that there is no 
evidence that the file ever existed in that location. The fact that it 
actually does still legitimately exist elsewhere is not a caveat - it is 
simply acting within standard behaviour.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Ideas for a feature implementation

[Erkki Seppala]

Austin S Hemmelgarn <ahferroin7@gmail.com> writes:

> Another thing that isn't listed there, that I would personally love to
> see is support for secure file deletion.

As discussed in the followups, it is difficult to do a really secure
deletion and a security-conscious person would opt to use encryption
instead. Ability to encrypt subvolumes or files would probably serve
those needs sufficiently, though it sounds like it could easily be a
longer task than 4 months.

-- 
  _____________________________________________________________________
     / __// /__ ____  __               http://www.modeemi.fi/~flux/\   \
    / /_ / // // /\ \/ /                                            \  /
   /_/  /_/ \___/ /_/\_\@modeemi.fi                                  \/

Re: Ideas for a feature implementation

[Konstantinos Skarlatos]

On 10/8/2014 10:21 μμ, Vimal A R wrote:
> Hello,
>
> I came across the to-do list at https://btrfs.wiki.kernel.org/index.php/Project_ideas and would like to know if this list is updated and recent.
>
> I am looking for a project idea for my under graduate degree which can be completed in around 3-4 months. Are there any suggestions and ideas to help me further?
Maybe help with Andrea Mazzoleni's New RAID library supporting up to six 
parities? It seems to be a great feature for btrfs.

https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html


>
> Thank you,
> Vimal
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Konstantinos Skarlatos

Re: Ideas for a feature implementation

[David Pottage]

On 12/08/14 12:00, Konstantinos Skarlatos wrote:
> Maybe help with Andrea Mazzoleni's New RAID library supporting up to 
> six parities? It seems to be a great feature for btrfs.
>
> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html

That would be very cool, but at present vanila RAID 5 or 6 does not work 
properly, so I think getting that fully working would be a better idea. 
(Unless it would make more sense to merge the whole lot into one bit of 
work where RAID 5 or 6 are just a special case of arbitrary parity level 
support).

At present, you can write RAID 5 or 6 data, but if anything goes wrong, 
btrfs cannot use the parity information to help you get your data back, 
so in general you are better off with RAID 1 or 10. Also, I don't think 
I/O done in parallel so you get no speed advantage from having multiple 
discs either.

-- 
David Pottage

Re: Ideas for a feature implementation

[Konstantinos Skarlatos]

On 13/8/2014 2:01 μμ, David Pottage wrote:
>
> On 12/08/14 12:00, Konstantinos Skarlatos wrote:
>> Maybe help with Andrea Mazzoleni's New RAID library supporting up to 
>> six parities? It seems to be a great feature for btrfs.
>>
>> https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg31735.html
>
> That would be very cool, but at present vanila RAID 5 or 6 does not 
> work properly, so I think getting that fully working would be a better 
> idea. (Unless it would make more sense to merge the whole lot into one 
> bit of work where RAID 5 or 6 are just a special case of arbitrary 
> parity level support).
>
> At present, you can write RAID 5 or 6 data, but if anything goes 
> wrong, btrfs cannot use the parity information to help you get your 
> data back, so in general you are better off with RAID 1 or 10. Also, I 
> don't think I/O done in parallel so you get no speed advantage from 
> having multiple discs either.
>
Yeah, thats one of the features I am waiting to get finished, because I 
already have 5 multi disk systems that i would prefer to migrate to 
RAID5/6 from RAID1/JBOD that they are now.

I dont know what is the best sequencing, I just think that these are 
great patches/features and its a pity for them to languish.


-- 
Konstantinos Skarlatos

Re: Ideas for a feature implementation

[Chris Murphy]

On Aug 13, 2014, at 5:01 AM, David Pottage <david@chrestomanci.org> wrote:

> At present, you can write RAID 5 or 6 data, but if anything goes wrong, btrfs cannot use the parity information to help you get your data back, so in general you are better off with RAID 1 or 10.

Btrfs RAID5/6 normally mounted will reconstruct data from parity if there's a data checksum mismatch indicating data block(s) are corrupt. It'll also reconstruct from parity in the case of degraded mounts, and when doing a balance. What's missing is scrub.


> Also, I don't think I/O done in parallel so you get no speed advantage from having multiple discs either.

Hmm I thought that was just single and raid1?


Chris Murphy