Questions concerning fstrim and online discard.

Questions concerning fstrim and online discard.

[Andreas Hartmann]

Hello!

Please, may I ask you some questions because I'm a little bit confused
about the behaviour of fstrim and online discard (ext4)?


I'm using the following configuration / partitioning with a SSD
(Controller: SF-2281; Corsair Force GT 240GB):

- dm_crypt: cryptsetup luksOpen - version 1.4.1, device-mapper
  version 1.02.75
- lvm version 2.02.96
- ext4, option discard; kernel version 3.4.11

All seems to work fine, I can't see any errors in messages or in dmesg
according trim / discard.

Now, I tested fstrim from util-linux version 2.19, and surprisingly got
this unexpected error:

fstrim: /: FITRIM ioctl failed: Operation not supported

Why do I get this error? Online discard always seemed to work fine (= I
didn't get any error).

I detected, that there is another necessary option to get a
working fstrim:
cryptsetup luksOpen must be run with option --allow-discards.


Now my questions:
- Did online discard work without --allow-discards? If not, why didn't I
  get any error message?
- Is there a difference between online discard and fstrim, means,
  fstrim doesn't work without --allow-discards, but online discard does?
- fstrim / always says after each reboot:
  /: 1701429248 bytes was trimmed
  This can't be true as the root partition contains mainly root and
  nothing else (no /var or /usr or even /opt). There has been no
  changes in fs since last power on / off cycle and fstrim during the
  last cycle.
  Doesn't fstrim remember the already trimmed regions? Does it trim
  always the complete unused regions?
- Which trim should be used with a SF-2281 controller concerning
  endurance? fstrim or online discard?


Thank you very much for your advice!
Kind regards,
Andreas

Re: Questions concerning fstrim and online discard.

[Andreas Hartmann]

Hello Lukas!

I'm really happy about your detailed comments! Thanks a lot!

On Tue, 16 Oct 2012, Andreas Hartmann wrote: Lukáš Czerner wrote:
> On Thu, 4 Oct 2012, Andreas Hartmann wrote:
> 
>> Date: Thu, 4 Oct 2012 06:47:36 +0200
>> From: Andreas Hartmann <andihartmann@01019freenet.de>
>> To: util-linux@vger.kernel.org
>> Subject: Questions concerning fstrim and online discard.
>> Resent-Date: Mon, 15 Oct 2012 16:10:43 +0200
>> Resent-From: Karel Zak <kzak@redhat.com>
>> Resent-To: Lukas Czerner <lczerner@redhat.com>
>>
>> Hello!
>>
>> Please, may I ask you some questions because I'm a little bit confused
>> about the behaviour of fstrim and online discard (ext4)?
>>
>>
>> I'm using the following configuration / partitioning with a SSD
>> (Controller: SF-2281; Corsair Force GT 240GB):
>>
>> - dm_crypt: cryptsetup luksOpen - version 1.4.1, device-mapper
>>   version 1.02.75
> 
> Hi Andreas,
> 
> I hope that you realize that using discard with dm_crypt is not
> safe.

I know about this problem. My understanding is: trim usually writes 0 to
the free addresses, hence it is possible to see which addresses are used
and which are unused.

The SF-2281 controller seems not to write zero to the addresses, hence
the problem shouldn't be with this controller? Or did I got something wrong?

cat /sys/block/sda/queue/discard_zeroes_data
0

> 
>> - lvm version 2.02.96
>> - ext4, option discard; kernel version 3.4.11
>>
>> All seems to work fine, I can't see any errors in messages or in dmesg
>> according trim / discard.
> 
> The question is what kernel version do you use ?

3.4.11 :-)

> Currnently we will
> ignore all errors coming from the online discard including the
> situation where the device does not support discard.
> 
> This is not exactly ideal, because we should really inform user that
> there was an error.

Yes, this would be really important!

> However we should _NOT_ inform user about every
> discard request failed with EOPNOTSUPP because we can have
> multi-disk devices where some of the devices support discard and
> some of the does not, this is perfectly valid situation and kernel
> should not scream.

Once at the moment the device is mounted would be enough.

> Nevertheless I am going to change things so that we inform use about
> problem (other than EOPNOTSUPP). And maybe warn once a day that
> part, or whole device does not support discard at all - I think that
> this should not hurt.
> 
>>
>> Now, I tested fstrim from util-linux version 2.19, and surprisingly got
>> this unexpected error:
>>
>> fstrim: /: FITRIM ioctl failed: Operation not supported
> 
> Actually kernel version is what matters.
> 
>>
>> Why do I get this error? Online discard always seemed to work fine (= I
>> didn't get any error).
> 
> Those errors probably got silently dropped so you did not know about
> them.

But if fstrim works, online discard will work, too? Or is this
conclusion wrong (ext4)?

[....]

> I hope it helps.

Yes, it really helped a lot!


Kind regards,
Andreas Hartmann

Re: Questions concerning fstrim and online discard.

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4396 bytes --]

On Tue, 16 Oct 2012, Andreas Hartmann wrote:

> Date: Tue, 16 Oct 2012 18:28:05 +0200
> From: Andreas Hartmann <andihartmann@01019freenet.de>
> To: Lukáš Czerner <lczerner@redhat.com>
> Cc: Andreas Hartmann <andihartmann@01019freenet.de>,
>     util-linux@vger.kernel.org, Karel Zak <kzak@redhat.com>
> Subject: Re: Questions concerning fstrim and online discard.
> 
> Hello Lukas!
> 
> I'm really happy about your detailed comments! Thanks a lot!
> 
> On Tue, 16 Oct 2012, Andreas Hartmann wrote: Lukáš Czerner wrote:
> > On Thu, 4 Oct 2012, Andreas Hartmann wrote:
> > 
> >> Date: Thu, 4 Oct 2012 06:47:36 +0200
> >> From: Andreas Hartmann <andihartmann@01019freenet.de>
> >> To: util-linux@vger.kernel.org
> >> Subject: Questions concerning fstrim and online discard.
> >> Resent-Date: Mon, 15 Oct 2012 16:10:43 +0200
> >> Resent-From: Karel Zak <kzak@redhat.com>
> >> Resent-To: Lukas Czerner <lczerner@redhat.com>
> >>
> >> Hello!
> >>
> >> Please, may I ask you some questions because I'm a little bit confused
> >> about the behaviour of fstrim and online discard (ext4)?
> >>
> >>
> >> I'm using the following configuration / partitioning with a SSD
> >> (Controller: SF-2281; Corsair Force GT 240GB):
> >>
> >> - dm_crypt: cryptsetup luksOpen - version 1.4.1, device-mapper
> >>   version 1.02.75
> > 
> > Hi Andreas,
> > 
> > I hope that you realize that using discard with dm_crypt is not
> > safe.
> 
> I know about this problem. My understanding is: trim usually writes 0 to
> the free addresses, hence it is possible to see which addresses are used
> and which are unused.

This is not exactly right. TRIM does not write anything to the
device, but you can read zeroes (or some other values, see bellow) when
reading previously trimmed blocks. The reason being that when when
it's tirmmed firmware does not actually need to read data from the flash.

> 
> The SF-2281 controller seems not to write zero to the addresses, hence
> the problem shouldn't be with this controller? Or did I got something wrong?
> 
> cat /sys/block/sda/queue/discard_zeroes_data
> 0

That's just one case. IIRC the device can return zeores after trim (which
will be advertised through sysfs interface), some other deterministic data
or pseudorandom data. The device would not be able to always return what
has been there before simply because those blocks might have already been
reused in wear levelling process, so it has to be substituted. And when it
comes to cryptography, all those options are bad.

> 
> > 
> >> - lvm version 2.02.96
> >> - ext4, option discard; kernel version 3.4.11
> >>
> >> All seems to work fine, I can't see any errors in messages or in dmesg
> >> according trim / discard.
> > 
> > The question is what kernel version do you use ?
> 
> 3.4.11 :-)
> 
> > Currnently we will
> > ignore all errors coming from the online discard including the
> > situation where the device does not support discard.
> > 
> > This is not exactly ideal, because we should really inform user that
> > there was an error.
> 
> Yes, this would be really important!
> 
> > However we should _NOT_ inform user about every
> > discard request failed with EOPNOTSUPP because we can have
> > multi-disk devices where some of the devices support discard and
> > some of the does not, this is perfectly valid situation and kernel
> > should not scream.
> 
> Once at the moment the device is mounted would be enough.
> 
> > Nevertheless I am going to change things so that we inform use about
> > problem (other than EOPNOTSUPP). And maybe warn once a day that
> > part, or whole device does not support discard at all - I think that
> > this should not hurt.
> > 
> >>
> >> Now, I tested fstrim from util-linux version 2.19, and surprisingly got
> >> this unexpected error:
> >>
> >> fstrim: /: FITRIM ioctl failed: Operation not supported
> > 
> > Actually kernel version is what matters.
> > 
> >>
> >> Why do I get this error? Online discard always seemed to work fine (= I
> >> didn't get any error).
> > 
> > Those errors probably got silently dropped so you did not know about
> > them.
> 
> But if fstrim works, online discard will work, too? Or is this
> conclusion wrong (ext4)?

Yes, if the file system supports both (which is the case for ext4).

> 
> [....]
> 
> > I hope it helps.
> 
> Yes, it really helped a lot!
> 

I am glad to help.

Thanks!
-Lukas

> 
> Kind regards,
> Andreas Hartmann

Re: Questions concerning fstrim and online discard.

[Andreas Hartmann]

Lukáš Czerner wrote:
> On Tue, 16 Oct 2012, Andreas Hartmann wrote:
> 
>> Date: Tue, 16 Oct 2012 18:28:05 +0200
>> From: Andreas Hartmann <andihartmann@01019freenet.de>
[...]
>>> Hi Andreas,
>>>
>>> I hope that you realize that using discard with dm_crypt is not
>>> safe.
>>
>> I know about this problem. My understanding is: trim usually writes 0 to
>> the free addresses, hence it is possible to see which addresses are used
>> and which are unused.
> 
> This is not exactly right. TRIM does not write anything to the
> device, but you can read zeroes (or some other values, see bellow) when
> reading previously trimmed blocks. The reason being that when when
> it's tirmmed firmware does not actually need to read data from the flash.
> 
>>
>> The SF-2281 controller seems not to write zero to the addresses, hence
>> the problem shouldn't be with this controller? Or did I got something wrong?
>>
>> cat /sys/block/sda/queue/discard_zeroes_data
>> 0
> 
> That's just one case. IIRC the device can return zeores after trim (which
> will be advertised through sysfs interface), some other deterministic data
> or pseudorandom data. The device would not be able to always return what
> has been there before simply because those blocks might have already been
> reused in wear levelling process, so it has to be substituted. And when it
> comes to cryptography, all those options are bad.

I read a few articles about encryption with SSD. With linux / dm-crypt /
cryptseup luks, plausible deniability isn't given at all because of the
architecture of cryptsetup luks and the not completely crypted disk.

Are there any known successfully carried out attacks (= partition /
filesystem was decryptable by the attacker) on crypted partitions on
SSDs which would have been not successful without TRIM enabled or is it
(as of today :-)) more of theory?



Thanks,
kind regards,
Andreas Hartmann

Re: Questions concerning fstrim and online discard.

[Milan Broz]

On 10/17/2012 07:28 PM, Andreas Hartmann wrote:
> I read a few articles about encryption with SSD. With linux / dm-crypt /
> cryptseup luks, plausible deniability isn't given at all because of the
> architecture of cryptsetup luks and the not completely crypted disk.

You can have separated LUKS header (with recent cryptetup) and if you
have filled device with random, you can map another device with different
offset into it. So you get the same arch like "plausible" deniability
in truecrypt, just it need some script magic.
But not possible with TRIM obviously... (hidden disk, possibly mapped
to unused outer disk space, could be discarded).

> Are there any known successfully carried out attacks (= partition /
> filesystem was decryptable by the attacker) on crypted partitions on
> SSDs which would have been not successful without TRIM enabled or is it
> (as of today :-)) more of theory?

I don't think so.

Discarded blocks are kind of side channel, you get more info about
device (like I tried to show it here - I can detect filesystem
from cipherdevice pattern here
http://asalor.blogspot.cz/2011/08/trim-dm-crypt-problems.html

Also read FAQ
http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions#5._Security_Aspects

But without another weakness it should not lead to decryption
of data (and if attacker can repeatedly get snapshots of ciphertext
device, it can be problem even today).

I would like people to tink about the problem, assess the risks and
if discard is acceptable for their particular use case, no problem
switch TRIM on.

And the most common case - encrypted laptop to secure data against
random thief - should still work even with TRIM.
But targeted attack is something different. Really, you need
particular threat model to say if it is a problem.

Milan