Severe fdisk problem leading to data loss?

Severe fdisk problem leading to data loss?

[Marc MERLIN]

I recently lost 10H of work because fdisk did something very bad: it created
multiple partitions starting in the same place, and when I mkfs'ed sda2, it
overwrote enough of sda4 to trash it beyond repair as well as killed linux by
writing over its swap on sda3

This is what my partitions looked like before I lost my system:

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes                                                          
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors                                
Units = sectors of 1 * 512 = 512 bytes                                                                 
Sector size (logical/physical): 512 bytes / 512 bytes                                                  
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                      
Disk identifier: 0x0b8ccbaa                                                                            
                                                                                                       
   Device Boot      Start         End      Blocks   Id  System                                         
/dev/sda1            2048     1050623      524288   83  Linux                                          
/dev/sda2         1050624   105908223    52428800    0  Empty                                          
/dev/sda3         1050624    34605055    16777216   82  Linux swap / Solaris                           
/dev/sda4        34605056  1953525167   959460056   83  Linux        

Sure enough, it's reproduceable.
I'm kind of very late and should go to bed (doing a full restore of my laptop now from 
backup). Would you be able to pass this to the right folks?

gandalfthegreat:~# dpkg -l util-linux
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-============================================
ii  util-linux          2.20.1-5.5     amd64          Miscellaneous system utilities
gandalfthegreat:~# fdisk -v
fdisk (util-linux 2.20.1)
gandalfthegreat:~# 


(I'm using -H32 -S32 here, but last time, I ran fdisk without them)
gandalfthegreat:/var/tmp# fdisk -H32 -S32 /dev/sda

Command (m for help): p

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
32 heads, 32 sectors/track, 1907739 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0b8ccbaa

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     1050623      524288   83  Linux
/dev/sda2         1050624   105908223    52428800    0  Empty

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 3): 
Using default value 3
First sector (1050624-1953525167, default 1050624): 105908224 << eek default is bad
Last sector, +sectors or +size{K,M,G} (105908224-1953525167, default 1953525167): +16G

Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Selected partition 4
First sector (1050624-1953525167, default 1050624):   << eek default is bad again
Using default value 1050624
Last sector, +sectors or +size{K,M,G} (1050624-105908223, default 105908223): 
Using default value 105908223

Command (m for help): p

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
32 heads, 32 sectors/track, 1907739 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0b8ccbaa

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1            2048     1050623      524288   83  Linux
/dev/sda2         1050624   105908223    52428800    0  Empty
/dev/sda3       105908224   139462655    16777216   83  Linux
/dev/sda4         1050624   105908223    52428800   83  Linux

This looks like a severe bug. 
Kernel was 3.11 if that helps.

Disk is:
Device Model:     Samsung SSD 840 EVO 1TB
Serial Number:    S1D9NEAD934600N
LU WWN Device Id: 5 002538 85009a8ff
Firmware Version: EXT0BB0Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
ATA Standard is:  ATA-8-ACS revision 4c

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Severe fdisk problem leading to data loss?

[Karel Zak]

On Sun, Nov 24, 2013 at 05:52:26AM -0800, Marc MERLIN wrote:
> Disk /dev/sda: 1000.2 GB, 1000204886016 bytes                          
> 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
> Units = sectors of 1 * 512 = 512 bytes                                
> Sector size (logical/physical): 512 bytes / 512 bytes                
> I/O size (minimum/optimal): 512 bytes / 512 bytes                   
> Disk identifier: 0x0b8ccbaa                                        
>                                                                   
>    Device Boot      Start         End      Blocks   Id  System   
> /dev/sda1            2048     1050623      524288   83  Linux   
> /dev/sda2         1050624   105908223    52428800    0  Empty  
                                                      ^^^^^^^^^^

Congratulation, you found very old fdisk bug as well as GNU Parted bug :-)

The problem is that the partition has no set partition type, and
fdisk in some situations interprets such partition as unused. 
 
So it lists the partition (print command), but the "add new
partition" command interprets this partition as unused.

> Command (m for help): n
> Partition type:
>    p   primary (1 primary, 0 extended, 3 free)
>    e   extended
> Select (default p): p
> Partition number (1-4, default 3): 
> Using default value 3
> First sector (1050624-1953525167, default 1050624): 105908224 << eek default is bad

 Yep.


It seems that GNU Parted is more consistent, it ignores the partition
at all :-), for example:

# fdisk -l /dev/sdb
...
Device    Boot     Start       End Blocks  Id System
/dev/sdb1           2048     22527  10240  83 Linux
/dev/sdb2          22528    227327 102400   0 Empty


# parted /dev/sdb print
...
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  11.5MB  10.5MB  primary


This is definitely bug too, because Linux does not care about
partition type and kernel is able to use this partition:

# lsblk /dev/sdb
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   0  600M  0 disk 
├─sdb1   8:17   0   10M  0 part 
└─sdb2   8:18   0  100M  0 part 


Note, don't use -S and -H, fdisk does not care about CHS by default
(you have to enable obsolete DOS mode ('c' command)).

I'm going to make fdisk more robust and prepare also a patch for GNU
parted.

Thanks!
    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Severe fdisk problem leading to data loss?

[Marc MERLIN]

On Mon, Nov 25, 2013 at 11:31:24AM +0100, Karel Zak wrote:
> >    Device Boot      Start         End      Blocks   Id  System   
> > /dev/sda1            2048     1050623      524288   83  Linux   
> > /dev/sda2         1050624   105908223    52428800    0  Empty  
> 
> Congratulation, you found very old fdisk bug as well as GNU Parted bug :-)
> 
> The problem is that the partition has no set partition type, and
> fdisk in some situations interprets such partition as unused. 
  
Aaah, I see.
If you wonder why I did that by the way, it's good to set aside portion
of an SSD and never use it. This leaves room for the garbage collector.

> So it lists the partition (print command), but the "add new
> partition" command interprets this partition as unused.

Understood.

> This is definitely bug too, because Linux does not care about
> partition type and kernel is able to use this partition:
 
It is indeed.

> Note, don't use -S and -H, fdisk does not care about CHS by default
> (you have to enable obsolete DOS mode ('c' command)).
 
Thanks for that. It was an old command I had to make sure I would get
big 64K block alignment for my SSD.

> I'm going to make fdisk more robust and prepare also a patch for GNU
> parted.

Thanks for your reply and the fixes.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Severe fdisk problem leading to data loss?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/25/2013 5:31 AM, Karel Zak wrote:
> This is definitely bug too, because Linux does not care about 
> partition type and kernel is able to use this partition:

Isn't the bug actually in the kernel, not parted?  An ID of zero means
the slot is unused, so it shouldn't be paying any attention to the
start and length.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSlgW3AAoJEI5FoCIzSKrwOiUIAIeaV+yAJZ4kDiU6OZU/jCwp
xIHnsyZ7Rg6XvaNBzgKl0RNuKx9ZvrcKJi2o/bmMr+k6dGl/vWO9/TbDfm5bP7gK
Z3Pt1GNbrPlXNwD0CQFoXbLi478vOcLG43/BfyISwc89BgKVosKFfVoeIL3YVFvM
okUJ9/W0Conasor9XXgT6xNdhEKelM80mWiNGzpUiruespVYO3AiDYp/SQvyfwyt
rAifX3tFNg5OULxL7DQSPlhpaknpnFwDB6wGzFW1ynR6vt+v/l3YBKAnczueMtex
X19qe4UiNkqHXtlCIvYI3W+MbasCbN/j/dtV7VcZfjvuBJEYX4uzRCq2eAcKxl0=
=+lAu
-----END PGP SIGNATURE-----

Re: Severe fdisk problem leading to data loss?

[Karel Zak]

On Wed, Nov 27, 2013 at 09:46:15AM -0500, Phillip Susi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 11/25/2013 5:31 AM, Karel Zak wrote:
> > This is definitely bug too, because Linux does not care about 
> > partition type and kernel is able to use this partition:
> 
> Isn't the bug actually in the kernel, not parted?  An ID of zero means
> the slot is unused, so it shouldn't be paying any attention to the
> start and length.

 After 20 years it's feature ;-) as I'm almost sure that we cannot fix
 it and disable all partition where is no the ID. 
 
 And Linux kernel have never cared about partition type (this is not
 specific to MBR, the same behaviour we have for Sun, SGI, MAC, BSD,
 ..), only PT where we care is GPT (zero GUID is ignored).

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Severe fdisk problem leading to data loss?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/27/2013 9:58 AM, Karel Zak wrote:
> After 20 years it's feature ;-) as I'm almost sure that we cannot
> fix it and disable all partition where is no the ID.

While that's cute and all, I think we both know it isn't really true ;)

> And Linux kernel have never cared about partition type (this is
> not specific to MBR, the same behaviour we have for Sun, SGI, MAC,
> BSD, ..), only PT where we care is GPT (zero GUID is ignored).

True, the type does not matter... so long as it is !0.  Windows will
consider such an entry unused and happily allow you to create a new
partition using that space, so we certainly don't want to allow people
to make such a partition in fdisk, and the kernel really should ignore
it too.  To do otherwise will lead people to lose data when Windows
thinks the entry is unused.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSljkQAAoJEI5FoCIzSKrwLLcIAJSimXfgq9QaHGzijwFUaMh6
gaqYX+Q5vW8L4zla2+14Fz6iO+mMr25iADwCLJPVsCTFSvjXOviRrL8010uTGKDv
c3/mOmzLZbVLJcrIguWAqQmv2GjWUpmcNLrD2eWUYUwn/v+u0GS3wN7raeycmLoN
rSi885Hpw6ODSghTnYSJSIvV+ExI8eavDq62giLWVWUScHwFR1YA3sPBo8PssNb5
oz088bO/x6k1GiXWNxfI7XbSbpcV7FHSvAvwKtQ0UZpVkE2vBRp+kDzLnx6C4nOd
iXJRE/84WzyqjYbkn1HF6GOs/mWQe1okkNRalgSObaJQ47cZ4RCxHHT2fmehjow=
=cQks
-----END PGP SIGNATURE-----

Re: Severe fdisk problem leading to data loss?

[Curtis Gedak]

On 13-11-27 11:25 AM, Phillip Susi wrote:
> On 11/27/2013 9:58 AM, Karel Zak wrote:
>> After 20 years it's feature ;-) as I'm almost sure that we cannot
>> fix it and disable all partition where is no the ID.
> While that's cute and all, I think we both know it isn't really true ;)

Unfortunately at least one company produces devices that have partitions 
with ID set to 0, yet these same partitions contain data.

An Apple iPod Shuffle is such an example as shown in the following fdisk 
output:

user@debian:~$ sudo fdisk -l /dev/sde
Note: sector size is 2048 (not 512)

Disk /dev/sde: 8120 MB, 8120172544 bytes
250 heads, 62 sectors/track, 255 cylinders
Units = cylinders of 15500 * 2048 = 31744000 bytes

    Device Boot      Start         End      Blocks   Id System
/dev/sde1               1           3       92876    0 Empty
/dev/sde2               4         255     7812000    b  W95 FAT32

This has lead some people, who wish to re-partition such a device, to 
use fdisk to temporarily set the ID to a non-zero value.  Then they can 
edit the partitions.  When the partitioning tasks are complete they use 
fdisk to set the ID back to zero again.

By doing this work-around, data loss can be avoided.

As such this "feature" of fdisk has been used and described on the 
Internet.  For example see the following post:
http://dreamlayers.blogspot.ca/2012/01/editing-ipod-partitions-with-gparted.html

Regards,
Curtis

Re: Severe fdisk problem leading to data loss?

[Karel Zak]

On Wed, Nov 27, 2013 at 01:25:20PM -0500, Phillip Susi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 11/27/2013 9:58 AM, Karel Zak wrote:
> > After 20 years it's feature ;-) as I'm almost sure that we cannot
> > fix it and disable all partition where is no the ID.
> 
> While that's cute and all, I think we both know it isn't really true ;)

:-)

> > And Linux kernel have never cared about partition type (this is
> > not specific to MBR, the same behaviour we have for Sun, SGI, MAC,
> > BSD, ..), only PT where we care is GPT (zero GUID is ignored).
> 
> True, the type does not matter... so long as it is !0.  Windows will
> consider such an entry unused and happily allow you to create a new
> partition using that space, so we certainly don't want to allow people
> to make such a partition in fdisk, and the kernel really should ignore
> it too.  To do otherwise will lead people to lose data when Windows
> thinks the entry is unused.

 windows bug? :-)

 Seriously, maybe it's fine to ignore the partition with the zero type
 by kernel, but I don't think it's good way for fdisk-like applications.
 
 IMHO it's better to force users to delete the partition than rely on
 the flag. See this thread, user lost data because somehow set the
 type to zero and fdisk reused the space. 
 
 From my point of view all partitions with non-zero size should be 
 visible by fdisk (or parted) independently on flags or types.
    
 I'll probably add a warning to fdisk to inform users about the zero 
 type problem.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Severe fdisk problem leading to data loss?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/27/2013 2:58 PM, Curtis Gedak wrote:
> Unfortunately at least one company produces devices that have
> partitions with ID set to 0, yet these same partitions contain
> data.
> 
> An Apple iPod Shuffle is such an example as shown in the following
> fdisk output:

Wow, what is in there?  And more hilariously, what happens when you
plug the thing into a Windows machine and try to create a partition in
the "free space"?

Also, I thought iPods don't use the USB MASS STORAGE protocol so they
don't show up as a block device at all; you have to use iTunes to
access them.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSllFaAAoJEI5FoCIzSKrw+lQIAKl7REeobwpPHtZrxbQztb67
SdgQScZUPy93Mi6qsnW4n6v090iIceo+g1HYnVLzzevNPAcJJRYQ1ZdKhpzgEPZe
OQFN8m1WnAKogCa1WsbAPtAKgDDaZnlLhAM3XVT4S2Yz5QZkx8EV5lVvTazo2w9X
ZhtpO1c8kvqr9P7y9EcDWnOiWWJRaXsUbqzsQ6GdtG+NtobDsheMPC8zsa0nWPn7
gjBMYWHuaVxJTuGQo2yyRPovpOOIxNYrd9fBTybI4np18a6dFD2cxPnipS/CwBPr
B4n5/YWKVf3M1jXZYPlUdlWrZVifCrO0NOkJ+/nlzIa/q71xRsGV/DhKbutA/Kg=
=7API
-----END PGP SIGNATURE-----

Re: Severe fdisk problem leading to data loss?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/27/2013 3:07 PM, Karel Zak wrote:
> Seriously, maybe it's fine to ignore the partition with the zero
> type by kernel, but I don't think it's good way for fdisk-like
> applications.
> 
> IMHO it's better to force users to delete the partition than rely
> on the flag. See this thread, user lost data because somehow set
> the type to zero and fdisk reused the space.

Only because fdisk let them set it to zero in the first place ;)

> From my point of view all partitions with non-zero size should be 
> visible by fdisk (or parted) independently on flags or types.

I don't like the idea of showing a different view of the partition
table than DOS/Windows; that way lies suffering.  I can maybe see
complaining that the entry is invalid and letting you change the type
to correct the problem ( assuming the start and length don't put it
outside the disk or overlapping other partitions ), but it should be
very clear that this is a broken entry.

> I'll probably add a warning to fdisk to inform users about the zero
>  type problem.

It already does warn you when you set the type to 0.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSllPcAAoJEI5FoCIzSKrw+6YH/AlZ34QNORe1YxXKAT3g02Ro
YMCVnS8iQdyw15Sj7zJDnl8ZVKkq87jlB/9HuHTrseMSdepK4pt38Gu9nkNXeFiU
7uyk5RLL9C6E90BE1GuV2FT0e02jLUS0BJpylMPFDSdmtNe7/3xVMOlxqkRGycFQ
CF0C2wGV878/pFje4XQbn48WGysVL+BN6PbjdWZZaOqAlldx154la+oYl9h+JkpO
KAfPZdtjIe/q9zXY4hQJzh7VIWgL/vf27/XYnzAXaxBxAiugdUMLinUQJ0j5TwB5
Ju8rN9P9RgQ9SAA6PNTfi5nWYPk1d6brTklQFCHdrGyJ54Pwdm7DKWOGaWxdtqQ=
=rHzX
-----END PGP SIGNATURE-----

Re: Severe fdisk problem leading to data loss?

[Curtis Gedak]

On 13-11-27 01:08 PM, Phillip Susi wrote:
> On 11/27/2013 2:58 PM, Curtis Gedak wrote:
>> Unfortunately at least one company produces devices that have
>> partitions with ID set to 0, yet these same partitions contain
>> data.
>>
>> An Apple iPod Shuffle is such an example as shown in the following
>> fdisk output:
> Wow, what is in there?  And more hilariously, what happens when you
> plug the thing into a Windows machine and try to create a partition in
> the "free space"?
>
> Also, I thought iPods don't use the USB MASS STORAGE protocol so they
> don't show up as a block device at all; you have to use iTunes to
> access them.

IIRC, the partition with ID = 0 contains the firmware for the Apple 
iPod.  The device is recognized in GNU/Linux, and files can be copied to 
and from the FAT partition.  However, since the iPod firmware uses a 
database to keep track of songs, only songs transferred to the device 
using "iPod aware" software will be seen by the iPod music player.  
Amarok is an example of a native GNU/Linux application that is iPod aware.

IMHO Apple's choice of using an ID of zero for the firmware partition is 
a poor decision.  With that being said, these devices with this strange 
ID setting do exist in the real world.

Curtis

Re: Severe fdisk problem leading to data loss?

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/27/2013 3:23 PM, Curtis Gedak wrote:
> IMHO Apple's choice of using an ID of zero for the firmware
> partition is a poor decision.  With that being said, these devices
> with this strange ID setting do exist in the real world.

So it doesn't bother the iPod if you change the id?

I'm thinking parted should throw an exception noting the broken
partition and offer to fix it by changing the id, or deleting it.
Changing the id would have the added benefit of keeping Windows from
overwriting it.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSll6EAAoJEI5FoCIzSKrwPCIH/2wSSKh/XT5J5fSpuqEG4NVq
+n/rhl8WpgfQhVjoAJbzoR9T6loDozdRXiqtunQ0kzGb8qreTiKmJJGMf/writiN
8PaFFyX+ga8geO6RALiTLp5yzNTUjUn8VGsaWwWozMwp5HsUte2lzVVcDPdKfY43
CBt8jXSXQxHyJlz2xbFlR2qo6/iGeUrYSI6rpoqMRnN6WU4KDBuZeJ3ByXg+Cli2
bFEGTr0B8lAJ3KSOWWBhOjSVpP22HypaTXI+xPth5LV4/AlbOuIZGuLk/uZSBnQF
ubv4xsodXfXcr97ZODUIstshKBS7pdVa33D2PE+4Yd0fOds0jWCSrKoubamIvCM=
=8DRl
-----END PGP SIGNATURE-----

Re: Severe fdisk problem leading to data loss?

[Curtis Gedak]

On 13-11-27 02:05 PM, Phillip Susi wrote:
> So it doesn't bother the iPod if you change the id?
>
> I'm thinking parted should throw an exception noting the broken
> partition and offer to fix it by changing the id, or deleting it.
> Changing the id would have the added benefit of keeping Windows from
> overwriting it.

I believe that you have to change the ID back to zero for the iPod to 
function properly.  At least that is what all of the instructions I've 
seen on the Internet indicate.

Re: Severe fdisk problem leading to data loss?

[Karel Zak]

On Wed, Nov 27, 2013 at 03:19:40PM -0500, Phillip Susi wrote:

> > I'll probably add a warning to fdisk to inform users about the zero
> >  type problem.
> 
> It already does warn you when you set the type to 0.

 Unfortunately not in v2.24, fixed now.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Severe fdisk problem leading to data loss?

[Ángel González]

On 27/11/13 22:05, Phillip Susi wrote:
> So it doesn't bother the iPod if you change the id?
>
> I'm thinking parted should throw an exception noting the broken
> partition and offer to fix it by changing the id, or deleting it.
> Changing the id would have the added benefit of keeping Windows from
> overwriting it.

Even if it worked and there were no known devices needing it at 0, it 
should be optional and allow to change it back. I have already faced a 
similar problem, having to jump through hoops, when restoring a 
partition layout and needing to set a "partition" to id 5 (extended).