[dm-crypt] `cryptsetup luksFormat` fails: "Cannot wipe header on device"

[Tom Roche <Tom_Roche@pobox.com>]

summary: LUKS newbie wants to LUKS/LVM2 on a dualboot, but several attempts to `cryptsetup luksFormat` the target partition have failed.

details:

(Apologies if this is a FAQ, but I'm not seeing answers from "the FAQ"

http://code.google.com/p/cryptsetup/wiki/FrequentlyAskedQuestions

or DDGing or googling the Subject above, as well as differing combinations of its terms.)

I have a laptop that came with Windows, which I previously dualbooted with a Debian Linux (LMDE), and on which I previously experimented with LUKS. It currently has

$ sudo fdisk -l /dev/sda
> Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)

> Disk /dev/sda: 500.1 GB, 500107862016 bytes
> 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0xce0b2a49

>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1            2048    34818047    17408000   27  Hidden NTFS WinRE
> /dev/sda2   *    34818048   239618047   102400000    7  HPFS/NTFS/exFAT
> /dev/sda3       239618048   240642047      512000   83  Linux
> /dev/sda4       240642048   976773119   368065536    5  Extended

I'd like to keep the first 2 partitions (i.e., their current contents should be kept as-is):

- /dev/sda1 = OEM diagnostics
- /dev/sda2 = OEM Windows

and redo the latter partitions (i.e., their current contents can be lost):

+ /dev/sda3 = Linux boot
+ /dev/sda4 = to be LVM2-manageable, LUKS-encrypted

I'm now experimenting with PePa's script for installing LMDE with LUKS and LVM2

http://j.mp/makelmdescript

which I've copied to a git repo

https://bitbucket.org/tlroche/install_resizable_encrypted_lmde

to facilitate better collaboration, extension, etc. I've added the script

https://bitbucket.org/tlroche/install_resizable_encrypted_lmde/raw/HEAD/install_LMDE_plus_LUKS_LVM2.sh

to a LiveUSB installer of LMDE-201403 (the latest), which boots and installs correctly: i.e., I have used that LiveUSB for another install (which appears good), and it boots the box on which I want to install the LMDE/LUKS/LVM2 combination. Once booted, I can

1. open a terminal (to bash)
2. run `cryptsetup benchmark`
3. open the script in an editor
4. `sudo -i` to become root
5. start running lines from the script (to learn more about it)

My problem is when I first start try to encrypt /dev/sda4:

# cryptsetup isLuks /dev/sda4 # null response
# echo -e "cryptsetup isLuks==$?"
> cryptsetup isLuks==1
# cryptsetup luksFormat --cipher=serpent-xts-plain64 --key-size=256 --hash=sha256 /dev/sda4
>
> WARNING!
> ========
> This will overwrite data on /dev/sda4 irrevocably.
>
> Are you sure? (Type uppercase yes): YES
> Enter passphrase:
> Verify passphrase:
> Cannot wipe header on device /dev/sda4.

How to fix? Some things I've tried (knowing almost nothing about LUKS or dm-crypt):

1. different cipher=aes-xts-plain64 (though `cryptsetup benchmark` shows serpent running much faster on my hardware): no change (wasn't expecting one :-)

2. `wipefs -a /dev/sda4`, then rerun `cryptsetup luksFormat ...`: no change.

3. `dd if=/dev/zero of=/dev/sda4`, then rerun `cryptsetup luksFormat ...`: no change.

In addition to the general question (how to fix?) I'd also like to know more about the cause: is this problem related to

* the warning above?

> Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)

* the fact that I'm attempting to install to an extended partition. E.g., do I need to create a logical partition=/dev/sda5 inside the extended partition=/dev/sda4 ?

Your assistance is appreciated, Tom Roche <Tom_Roche@pobox.com>