[dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Martin Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

Hi all,

I've been cryptsetup and LUKS-encryption a lot in the past 5 years and the
only time I've really lost data is because I locked myself out of my linux
boot-partition during a kernel upgrade/initramfs-generation (still not sure
what happened, but I've learned now that I should ALWAYS have a backup
password instead of relying on a key which I lost). But the last 3-4 days
scared the shit out of me, as I took a complete backup of maybe 15 years of
data (5-6 TB), verified with md5sum that the copy also was ok - but in
order to have space for that (~5-6 TB) I deleted my previous unencrypted
backup, otherwise I had to buy a new disk and I trusted things were ok...
When I saw the md5sum on the backup disk was ok, I felt "safe" and wanted
to unlock my partition (just after deleting the unencrypted data, to have
space). I used cryptsetup on a Debian (proxmox)-installation where the
encrypted data was also stored and it worked before doing the backup. But
after making the backup I got something like:

*"mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
missing codepage or helper program, or other error*

At first I thought it was simple: I googled and found out I should use
e2fsck with an option to use another superblock. I read I should try the
first, reboot, then the next, reboot, etc - until I've went through them
all or until it works.... It didn't work - I was shocked, I thought I had
made the mistake of my life and lost all my data...

Nothing on google pointed me to the solution - I was screwed - I thought. I
think the problem happened because I was primarily using Debian / proxmox
(where the encrypted data is) with cryptsetup 2.1.0. But it might have
happened that just before doing the backup, I had "sshfs'ed" to my laptop
with Arch Linux, which is using cryptsetup 2.3.4. I think after opening the
LUKS-container with cryptsetup 2.3.4 (and closing it again after use, hence
probably forcing some things to be written), the encrypted data couldn't -
and cannot be opened with cryptsetup 2.1.0 anymore... At least this was/is
the solution for me: Don't use debian/proxmox with cryptsetup 2.1.0
anymore, instead use Arch Linux with version 2.3.4... I'm still terrified -
but tried to investigate the difference with luksDump (encrypted file). It
says LUKS is version 2, epoch 3, pbkdf2 is used, aes-xts-plain64 cipher
with 512 bits key - and everything is completely the same using both
luksDump-versions...

I tried to look in version history but am not into the details of the
difference in crypt-setup versions. Can anyone please tell why I got/get
this error using 2.1.0 and not with 2.3.4?

"mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
missing codepage or helper program, or other error

Questions that popup are:  Is there a fix? Is it recommended to stick with
2.3.4? Should I worry in the future?

I would be grateful to understand this better, to not screw up in the
future. Furthermore, I think or hope this could/can maybe help others, as I
didn't found this solution on google or any other place. I almost gave up
all hope and was prepared to delete the LUKS-container, before I thought I
would try this "long shot" - phew, I'm just happy I have my data back.
Please elaborate or explain the version difference together with your
comments, I would be very grateful for that, thanks!


Br,
M.

[-- Attachment #2: Type: text/html, Size: 3761 bytes --]

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Michael Kjörling]

On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
> I tried to look in version history but am not into the details of the
> difference in crypt-setup versions. Can anyone please tell why I got/get
> this error using 2.1.0 and not with 2.3.4?
> 
> "mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
> missing codepage or helper program, or other error

_If_ you're able to unlock the container using both versions, _but_
with one you are able to mount the file system within the container
and with the other you are unable to do so, then I would suggest
comparing the output of `file -s /dev/mapper/whatever` for the two
cases after unlocking. You could also try `cryptsetup luksDump
--dump-master-key` for each version of cryptsetup and comparing the
output, but be sure that you understand the consequences of the master
key being potentially compromised before you do so.

I'm sure someone will correct me if I'm wrong about this, but I am
_fairly_ certain that there is nothing in the luksOpen/luksClose code
that will change anything in the LUKS header, so something else must
be going on in your case.

Are you able to reproduce the behavior with a brand new container?
(You can use a plain file to hold a LUKS container.) If so, please
list the exact commands, in order, and the exact software versions
(kernel, cryptsetup, libcryptsetup) involved.

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
 “Remember when, on the Internet, nobody cared that you were a dog?”

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Milan Broz]

On 25/11/2020 19:01, Michael Kjörling wrote:
> On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
>> I tried to look in version history but am not into the details of the
>> difference in crypt-setup versions. Can anyone please tell why I got/get
>> this error using 2.1.0 and not with 2.3.4?
>>
>> "mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
>> missing codepage or helper program, or other error
> 
> _If_ you're able to unlock the container using both versions, _but_
> with one you are able to mount the file system within the container
> and with the other you are unable to do so, then I would suggest
> comparing the output of `file -s /dev/mapper/whatever` for the two
> cases after unlocking. You could also try `cryptsetup luksDump
> --dump-master-key` for each version of cryptsetup and comparing the
> output, but be sure that you understand the consequences of the master
> key being potentially compromised before you do so.

Yes, but the key is validated through LUKS key digest, if the validation fails,
it is never used for the mapping. I do not believe it is a wrong key problem.

Also compare "dmsetup table <device>" on both systems for LUKS active device.
It should produce exactly the same parameters (mainly offset, size, crypt options).

All this looks as a kernel issue unrelated to cryptsetup (note that you have different
kernels if you have different distros and the data decryption and fs handling is the kernel).

> I'm sure someone will correct me if I'm wrong about this, but I am
> _fairly_ certain that there is nothing in the luksOpen/luksClose code
> that will change anything in the LUKS header, so something else must
> be going on in your case.

For LUKS1 is it always true, LUKS2 have two metadata areas, there is autocorrection
in place if one area checksum is wrong (happens on the header load) but this is IMO
unrelated to this case.

(If I understand the mail correctly, the LUKS device can be still activated but
the filesystem is corrupted now? If not, we need --debug output from luksOpen command
to check what is wrong.)

m.

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Martin Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 8021 bytes --]

On Wed, Nov 25, 2020 at 7:34 PM Milan Broz <gmazyland@gmail.com> wrote:

> On 25/11/2020 19:01, Michael Kjörling wrote:
> > On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
> >> I tried to look in version history but am not into the details of the
> >> difference in crypt-setup versions. Can anyone please tell why I got/get
> >> this error using 2.1.0 and not with 2.3.4?
> >>
> >> "mount: wrong fs type, bad option, bad superblock on /dev/mapper/....,
> >> missing codepage or helper program, or other error


Michael, please reply (reply all) to me also, it seems I only received your
comments/message via Milan's reply (otherwise I guess I would have seen it
in the digest, I've turned digest off for now to ensure I receive all
replies in the coming days)...

>
>
> _If_ you're able to unlock the container using both versions, _but_
> > with one you are able to mount the file system within the container
> > and with the other you are unable to do so, then I would suggest
>

Right, sorry. This is correctly understood and this is what happens. It's
the mount-command that causes problems, not actually cryptsetup itself,
good to have this clarified...


> > comparing the output of `file -s /dev/mapper/whatever` for the two
> > cases after unlocking. You could also try `cryptsetup luksDump
>

I think you meant "file -Ls".

For both machines I get: /dev/mapper/TESTLUKS: Linux rev 1.0 ext4
filesystem data, UUID=d808dde4-2e57-48ed-b9e7-5c21948fabe6 (needs journal
recovery) (extents) (64bit) (large files) (huge files)

But I don't understand "needs journal recovery", could this have anything
to do with it?


> > --dump-master-key` for each version of cryptsetup and comparing the
> > output, but be sure that you understand the consequences of the master
> > key being potentially compromised before you do so.
>
> Yes, but the key is validated through LUKS key digest, if the validation
> fails,
> it is never used for the mapping. I do not believe it is a wrong key
> problem.
>
> Right, the key is ok - I can unlock, but can't mount, thanks Milan.


> Also compare "dmsetup table <device>" on both systems for LUKS active
> device.
> It should produce exactly the same parameters (mainly offset, size, crypt
> options).
>

Looks exactly similar - in both cases: 0 6291423232 crypt aes-xts-plain64
:64:logon:cryptsetup:265a5d99-4563-49e6-a5a5-7d7cab4f137e-d0 0 7:1 32768

All this looks as a kernel issue unrelated to cryptsetup (note that you
> have different
> kernels if you have different distros and the data decryption and fs
> handling is the kernel).
>

Right, that makes sense, I agree with you, sounds very much like a kernel
thing with "mount"... Proxmox/debian runs 5.4.65-1. Arch Linux runs
5.9.9... Oh yeah, and by the way, I actually did upgrade the proxmox/debian
to another/new kernel just before copying the backup to another disk. I can
see in my /boot-dir, that I used to run 5.4.55-1 and later 5.4.60-1, just
before these issue appeared, maybe there's a problem with the 5.4.65-1
kernel, which wasn't there in 5.4.60-1...

There's actually some talk about "journal superblock" here
https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.4.61 (and in
changelog for 5.4.62, but I do not understand that) - and my error message
was something about "mount: wrong fs type, bad option, bad superblock on
/dev/mapper/...., missing codepage or helper program... hmm.


> > I'm sure someone will correct me if I'm wrong about this, but I am
> > _fairly_ certain that there is nothing in the luksOpen/luksClose code
> > that will change anything in the LUKS header, so something else must
> > be going on in your case.
>
> For LUKS1 is it always true, LUKS2 have two metadata areas, there is
> autocorrection
> in place if one area checksum is wrong (happens on the header load) but
> this is IMO
> unrelated to this case.
>

Damn, it's really creepy... I actually didn't check/test mounting on the
Debian/proxmox-machine since after I spend a lot of time seeing that it
worked on Arch Linux (and at that time not on the
Debian/proxmox-machine)... But now it works on both machines (arch+debian)?
Weird... It was 2 days ago I had these problems... Are you sure this
autocorrection is unrelated? Let me refresh what I did, because luckily I
still have the complete logs:

So, when I tried to repair the LUKS-encrypted container, google said to me
that I should try using another superblock. Here's what happened on the
debian/proxmox machine:

root@proxmox:/mnt# mke2fs -n /dev/mapper/hugeZFSdata
mke2fs 1.44.5 (15-Dec-2018)
Found a atari partition table in /dev/mapper/hugeZFSdata
Proceed anyway? (y,N) y
Creating filesystem with 786430976 4k blocks and 196608000 inodes
Filesystem UUID: 6af28573-a3b9-43a1-9562-f9c990a87598
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544

And this didn't seem to work at all (I tried MANY TIMES + rebooted each
time):
-----
# e2fsck  -b 32768 -v  /dev/mapper/hugeZFSdata
e2fsck 1.44.5 (15-Dec-2018)
e2fsck: Bad magic number in super-block while trying to open
/dev/mapper/hugeZFSdata
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Found a atari partition table in /dev/mapper/hugeZFSdata
----

I remember that although I tried ALL the blocks I always got the same error
message about "Found a atari partition table in /dev/mapper/hugeZFSdata",
which made me think that the command failed. One post found via google said
I should try and ignore the message, just run it and reboot (to flush
something, I guess). I did reboot after around 1/2 of the times, but there
are 21 blocks to test and I didn't have patience to reboot after all 21
attempts... Maybe I was just lucky?

But I would *never* have tried the "work-around" I eventually - as a final
step - came up with, which is to SMB/CIFS/Samba-mount the encrypted
file/container to the Arch machine, where I mounted the folder and ran
"cryptsetup luksOpen ..." + mounted the dev/mapper/....-device
successfully...

Hmm... I'm still confused - I hope we find the explanation, so I know what
to do if it happens again. If it happens again, however, I'll run the
commands above and compare them again + hopefully write here, in case
there's a bug somewhere. But I think you're right, that it likely is some
kernel stuff with the mount-command. The "autocorrection" for the two
LUKS2-metadata areas, does that happen after a successful "mount"-command
or as part of the "cryptsetup luksOpen ..."-command?

It's weird that it seems to be working now, I think there's some
autocorrection going on (I did remember updating a few packages on the
debian-machine, but not much, chances are those upgrades, not even kernel
updates, didn't change this)... hmm.


> (If I understand the mail correctly, the LUKS device can be still
> activated but
> the filesystem is corrupted now? If not, we need --debug output from
> luksOpen command
> to check what is wrong.)
>
>  I'm very interested in hearing what you make out of the "The superblock
could not be read or does not describe a valid ext2/ext3/ext4 filesystem"
and the "Found a atari partition table in /dev/mapper/hugeZFSdata"-error
message I posted above, while I had the problem... I'm very afraid if the
problem happens again, damn, it was/is scary, got any ideas about what
could've happened? I'm really confused now :-)

Br,
M.

[-- Attachment #2: Type: text/html, Size: 10439 bytes --]

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Martin Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 3967 bytes --]

On Wed, Nov 25, 2020 at 7:34 PM Milan Broz <gmazyland@gmail.com> wrote:

> On 25/11/2020 19:01, Michael Kjörling wrote:
> > On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
> >> I tried to look in version history but am not into the details of the
> >> difference in crypt-setup versions. Can anyone please tell why I got/get
> >> this error using 2.1.0 and not with 2.3.4?
> --- snip ---


Just a small update: Things are much better now... After writing the last
reply, I had another kernel upgrade (on the debian/proxmox, where the data
is), am now running 5.4.73-1 and fsck is clean:

# e2fsck /dev/mapper/hugeData
e2fsck 1.44.5 (15-Dec-2018)
/dev/mapper/hugeData: clean, 1706487/196608000 files, 724924235/786427904
blocks

My issue sounds more and more like a kernel/mount/ext4-fs issue than
actually a LUKS/cryptsetup-issue, that is really great to hear... I checked
my dmesg, here on the debian/proxmox-system (for EXT4-fs, should've done
that along the way):

1414:[Tue Nov 24 00:29:03 2020] EXT4-fs (loop0): warning: mounting fs with
errors, running e2fsck is recommended
1415:[Tue Nov 24 00:29:03 2020] EXT4-fs (loop0): mounted filesystem with
ordered data mode. Opts: (null)
1449:[Tue Nov 24 00:34:06 2020] EXT4-fs (loop0): error count since last
fsck: 2
1450:[Tue Nov 24 00:34:06 2020] EXT4-fs (loop0): initial error at time
1601769126: ext4_journal_check_start:61
1451:[Tue Nov 24 00:34:06 2020] EXT4-fs (loop0): last error at time
1601769126: ext4_journal_check_start:61
1453:[Wed Nov 25 01:50:10 2020] EXT4-fs (loop0): error count since last
fsck: 2
1454:[Wed Nov 25 01:50:10 2020] EXT4-fs (loop0): initial error at time
1601769126: ext4_journal_check_start:61
1455:[Wed Nov 25 01:50:10 2020] EXT4-fs (loop0): last error at time
1601769126: ext4_journal_check_start:61
1456:[Wed Nov 25 19:44:14 2020] EXT4-fs (dm-13): recovery complete
1457:[Wed Nov 25 19:44:14 2020] EXT4-fs (dm-13): mounted filesystem with
ordered data mode. Opts: (null)
1465:[Wed Nov 25 19:48:13 2020] EXT4-fs (dm-13): mounted filesystem with
ordered data mode. Opts: (null)

I googled and the "recovery complete" line seems to be "journal recovery
after an unclean shutdown", I'm not sure what triggered this "recovery" (as
I remember it, I couldn't mount /dev/mapper/...) - but the LUKS-device was
mounted successfully on an Arch Linux VM via Samba/CIFS/SMB-network and via
a new cryptsetup-util.

This makes me think that I've got a new idea about what could've happened
(I hope not, I should definately know better - but with those EXT4-fs
entries, I begin to believe I could've had a weak moment and done something
stupid):
It could maybe be possible (cannot remember it) that I had a luksOpen'ed
the container, e.g. having /dev/mapper/whatever and maybe one of 2 things
happened:

1) Maybe /dev/mapper/whatever existed, but it wasn't mounted (to
/mnt/something), but the LUKS container also wasn't closed and maybe I had
started making the backup here...?
2) Maybe /dev/mapper/whatever existed and at the same time it was mounted
(to /mnt/something) - the worst scenario...

These 2 scenarios (the first most likely), might've been what happened -
otherwise I have absolutely no idea what can explain this (been using
cryptsetup for around 5 years, haven't had such a problem before)... I've
had a terrible experience, but have hopefully learned my lesson, for the
future.

If nothing else comes up, I think I'm gonna assume that I've had a weak
moment and maybe did umount, but perhaps I didn't luksClose'd before
starting the backup (scenario 1 above) and this I think could've corrupted
the ext4-fs, which for some reason I couldn't mount for some days and that
was (for a reason I don't understand) fixed when I mounted the FS on the
arch linux (newer kernel, maybe better ext4-fs auto-correction code)... In
any case: Thanks for the feedback and good work.

Br,
M

[-- Attachment #2: Type: text/html, Size: 4548 bytes --]

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Martin Jørgensen]

[-- Attachment #1: Type: text/plain, Size: 2921 bytes --]

On Thu, Nov 26, 2020 at 10:32 AM Martin Jørgensen <mjoerg@gmail.com> wrote:

> On Wed, Nov 25, 2020 at 7:34 PM Milan Broz <gmazyland@gmail.com> wrote:
>
>> On 25/11/2020 19:01, Michael Kjörling wrote:
>> > On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
>> >> I tried to look in version history but am not into the details of the
>> >> difference in crypt-setup versions. Can anyone please tell why I
>> got/get
>> >> this error using 2.1.0 and not with 2.3.4?
>> --- snip ---
>
>
> Just a small update: Things are much better now... After writing the last
> reply, I had another kernel upgrade (on the debian/proxmox, where the data
> is), am now running 5.4.73-1 and fsck is clean:
>

Update again: Just for the reference (about corrupted LUKS-volume during
transfer of some TB files): I had done nothing wrong (sorry for delayed
answer, but transferring many TB and md5sum on my lousy home-network to an
old Synology Diskstation device takes days). The culprit is that I didn't
use my harddisk volume with SATA-cables, but instead relied on a piece of
crap like this:
https://www.thermaltakeusa.com/thermaltake-blacx-hdd-docking-station.html
and my encrypted LUKS-volume was being copied via USB3 - I'll never do that
again (I trusted that thermaltake hdd docking station).

I had some conversations going on about my invalid md5sum file transfer (
https://bbs.archlinux.org/viewtopic.php?pid=1943811#p1943811 ), causing an
apparent corrupted LUKS-container and I'm thrilled that even though some
random bytes must've been written I think the LUKS-encrypted container
(containing ext4 filesystem must've repaired itself, self-healing or
something?) to the extent that most of it is still readable (although I
haven't checked it all, it's 3 TB of data, old documents, movies, music
etc). I'm surprised that I didn't lose all my data and that LUKS seems
pretty robust, even in the case like me, where I had corrupted data (my
problem had obviously nothing to do with version 2.1.0 vs 2.3.4, but
initially I couldn't read my data, i.e. couldn't mount the opened
LUKS-device mapper to /mnt/...)... I'm however betting that if the
LUKS-header was corrupt, I was doomed... I learned my lesson and will never
ever again use USB-3 for transferring my LUKS-containers. For the
reference, if anyone experiences the same problem I don't recommend using
usb at all for transferring LUKS-volumes. Thanks all LUKS/dm-crypt
developers for doing such a great job and sorry for me being so confused
about this, writing a few confusing mails, until I discovered the root
cause of the problem and the solution to prevent future problems (all
hardware, not even a software-problem, I had no idea, sorry about that but
now my faith in LUKS-encryption and the ability to resist even some random
bytes here and there, is stronger than ever before :-) )...

Br,
M

[-- Attachment #2: Type: text/html, Size: 3870 bytes --]

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Martin Jørgensen]

[-- Attachment #1.1: Type: text/plain, Size: 2921 bytes --]

On Thu, Nov 26, 2020 at 10:32 AM Martin Jørgensen <mjoerg@gmail.com> wrote:

> On Wed, Nov 25, 2020 at 7:34 PM Milan Broz <gmazyland@gmail.com> wrote:
>
>> On 25/11/2020 19:01, Michael Kjörling wrote:
>> > On 25 Nov 2020 18:30 +0100, from mjoerg@gmail.com (Martin Jørgensen):
>> >> I tried to look in version history but am not into the details of the
>> >> difference in crypt-setup versions. Can anyone please tell why I
>> got/get
>> >> this error using 2.1.0 and not with 2.3.4?
>> --- snip ---
>
>
> Just a small update: Things are much better now... After writing the last
> reply, I had another kernel upgrade (on the debian/proxmox, where the data
> is), am now running 5.4.73-1 and fsck is clean:
>

Update again: Just for the reference (about corrupted LUKS-volume during
transfer of some TB files): I had done nothing wrong (sorry for delayed
answer, but transferring many TB and md5sum on my lousy home-network to an
old Synology Diskstation device takes days). The culprit is that I didn't
use my harddisk volume with SATA-cables, but instead relied on a piece of
crap like this:
https://www.thermaltakeusa.com/thermaltake-blacx-hdd-docking-station.html
and my encrypted LUKS-volume was being copied via USB3 - I'll never do that
again (I trusted that thermaltake hdd docking station).

I had some conversations going on about my invalid md5sum file transfer (
https://bbs.archlinux.org/viewtopic.php?pid=1943811#p1943811 ), causing an
apparent corrupted LUKS-container and I'm thrilled that even though some
random bytes must've been written I think the LUKS-encrypted container
(containing ext4 filesystem must've repaired itself, self-healing or
something?) to the extent that most of it is still readable (although I
haven't checked it all, it's 3 TB of data, old documents, movies, music
etc). I'm surprised that I didn't lose all my data and that LUKS seems
pretty robust, even in the case like me, where I had corrupted data (my
problem had obviously nothing to do with version 2.1.0 vs 2.3.4, but
initially I couldn't read my data, i.e. couldn't mount the opened
LUKS-device mapper to /mnt/...)... I'm however betting that if the
LUKS-header was corrupt, I was doomed... I learned my lesson and will never
ever again use USB-3 for transferring my LUKS-containers. For the
reference, if anyone experiences the same problem I don't recommend using
usb at all for transferring LUKS-volumes. Thanks all LUKS/dm-crypt
developers for doing such a great job and sorry for me being so confused
about this, writing a few confusing mails, until I discovered the root
cause of the problem and the solution to prevent future problems (all
hardware, not even a software-problem, I had no idea, sorry about that but
now my faith in LUKS-encryption and the ability to resist even some random
bytes here and there, is stronger than ever before :-) )...

Br,
M

[-- Attachment #1.2: Type: text/html, Size: 3870 bytes --]

[-- Attachment #2: Type: text/plain, Size: 135 bytes --]

_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Milan Broz]

On 16/12/2020 01:06, Martin Jørgensen wrote:

> For the reference, if anyone experiences the same problem I don't recommend using usb at all for transferring LUKS-volumes.

Just a small addition:

This is really not LUKS specific, if you have failing HW data is going to be corrupted - just in the case of encrypted data
the error will usually propagate to whole sectors, so it is more visible to upper layer (fs).

And many external drivers or USB interfaces are really unreliable these days in long-term use...

If there is no problem with HW, LUKS can be easily used for external USB disks. In fact it is one of intended use case.

Anyway, thanks for letting us know where the problem was!

Milan
_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] cryptsetup versions: cryptsetup 2.3.4 vs. cryptsetup 2.1.0 - thought I had data corruption!?!

[Arno Wagner]

On Tue, Dec 22, 2020 at 14:21:20 CET, Milan Broz wrote:
> On 16/12/2020 01:06, Martin Jørgensen wrote:
> 
> > For the reference, if anyone experiences the same problem I don't recommend using usb at all for transferring LUKS-volumes.
> 
> Just a small addition:
> 
> This is really not LUKS specific, if you have failing HW data is going to
> be corrupted - just in the case of encrypted data the error will usually
> propagate to whole sectors, so it is more visible to upper layer (fs).

Pretty much. When I did a measurement for my PhD (a long time ago),
I needed to pull 15TB from tape (disk were up to 200GB at that time).
I found about 15 bit-errors in there on decompression that were not
noticed in any place before and likely would have gone unnoticed if
the data had not been compressed. Crypto is simimar in that it amplifies
bit-erors and suddenly you see them, even if they were there before.

> And many external drivers or USB interfaces are really unreliable these
> days in long-term use...

Yes. And often it is some chip-internal bus or memory that is not 
protected with checksums and fries a bit every few GB you pipe 
through there.

Regards,
Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier
_______________________________________________
dm-crypt mailing list
dm-crypt@saout.de
https://www.saout.de/mailman/listinfo/dm-crypt