Linear mode: array size is less than sum of sizes

Linear mode: array size is less than sum of sizes

[Alessandro Baretta]

Hi guys.

I have a bizarre problem. Hopefully someone will provide some insight.

I would like to wrap my NTFS Windows XP partition into a virtual hard
disk for use with virtualization tools such as XEN or KVM. In order to
turn a partition into a virtual hard disk I must prepend enough space
at the beginning to allow space for the MBR and the partition table.
The idea I have is to create a little partition (/dev/sda5) on the
tail of my hard drive and use the md driver in linear mode to prepend
this to the NTFS partition (/dev/sda1). The resulting md device should
appear to Windows as a real hard drive, with its own MBR and partition
table.

I am using the following command to create the md device:

mdadm --build /dev/md0 -l linear -n2 /dev/sda5 /dev/sda1

The md device gets created properly and can be used with fdisk to
create a partition table. However, the size of the md device, as
reported by fdisk,  does not match the sum of the size of the
constituent block devices, always as reported by fdisk.

Here is a table showing the start and end sectors and the computed and
reported sizes of the two constituent partitions, sda1 and sda5, and
of the md device.

block dev		reported size	start			end			computed size		delta
/dev/sda1		241797953			63	241798015	241797953		0
/dev/sda5			16002	491797908	491813909		16002		0
/dev/md0		241813888							241813955		-67

As you can see the computed size is 67 sectors less than the reported
size. What happened to those sectors? They are surely not used to
store superblocks, as the --build command does not create superblocks.
If they are used for the md driver's magic, do they all come from the
first device of the array or are they distributed on both? All in all,
how safe (or unsafe) is my approach to virtualizing Windows XP on my
Linux box?

Thank you in advance for any help you can provide.

Alex

Re: Linear mode: array size is less than sum of sizes

[John Robinson]

On 05/01/2010 18:21, Alessandro Baretta wrote:
> Hi guys.
> 
> I have a bizarre problem. Hopefully someone will provide some insight.
[...]
> As you can see the computed size is 67 sectors less than the reported
> size. What happened to those sectors?

At the very least md linear mode will round down the constituent devices 
to multiples of the rounding size, which is the same as the chunk size - 
see `man mdadm` for rounding size. The default 64K is 128 sectors, and 
probably explains what's happened to your 67 sectors - sda1 has its last 
65 sectors ignored, and sda5 2. You specify the chunk size in KiB not 
sectors so I don't know how you're going to fix this, and I've a feeling 
performance would be woeful even if you could have a 1-sector chunk size.

> They are surely not used to
> store superblocks, as the --build command does not create superblocks.
> If they are used for the md driver's magic, do they all come from the
> first device of the array or are they distributed on both? All in all,
> how safe (or unsafe) is my approach to virtualizing Windows XP on my
> Linux box?

I don't really know how safe it is, but I don't like your 16002-sector 
thingy. If you're going to do this I can't help thinking you should 
duplicate the real partition table, showing the partition starting on 
sector 63, so you want 63 extra sectors, not 16002. You might do it 
using a file on your filesystem via loopback instead of messing about 
with a silly 1-cylinder partition.

> Thank you in advance for any help you can provide.

Good luck and let me/us know how you get on!

Cheers,

John.

Re: Linear mode: array size is less than sum of sizes

[John Robinson]

On 05/01/2010 20:06, John Robinson wrote:
> On 05/01/2010 18:21, Alessandro Baretta wrote:
>> Hi guys.
>>
>> I have a bizarre problem. Hopefully someone will provide some insight.
> [...]
>> As you can see the computed size is 67 sectors less than the reported
>> size. What happened to those sectors?
> 
> At the very least md linear mode will round down the constituent devices 
[...]

Try playing with dm instead of md - `man dmsetup` suggests it may be 
able to do a linear concatenation in single-sector amounts...

Cheers,

John.

Re: Linear mode: array size is less than sum of sizes

[Alessandro Baretta]

Hi John,

First of all thanks for your help.

On Tue, Jan 5, 2010 at 9:06 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
>
>> As you can see the computed size is 67 sectors less than the reported
>> size. What happened to those sectors?
>
> At the very least md linear mode will round down the constituent devices to
> multiples of the rounding size, which is the same as the chunk size - see
> `man mdadm` for rounding size. The default 64K is 128 sectors, and probably
> explains what's happened to your 67 sectors - sda1 has its last 65 sectors
> ignored, and sda5 2. You specify the chunk size in KiB not sectors so I
> don't know how you're going to fix this, and I've a feeling performance
> would be woeful even if you could have a 1-sector chunk size.

That's exactly it! I thought that the 64k chunks were only relevant
for RAID1 and greater. I am losing 65 sectors on /dev/sda1 and 2 on
/dev/sda5. I have checked that the the first 64k of /dev/sda1 are
equal to the 64k of /dev/md0 following the first 16000 sectors. I
could try to resize the NTFS partition up by 63 sectors, hopefully
without clobbering it beyond repair, and map sectors 16001 to the end
of the device to the first partition.

>> [...] All in all,
>> how safe (or unsafe) is my approach to virtualizing Windows XP on my
>> Linux box?
>
> I don't really know how safe it is, but I don't like your 16002-sector
> thingy. If you're going to do this I can't help thinking you should
> duplicate the real partition table, showing the partition starting on sector
> 63, so you want 63 extra sectors, not 16002. You might do it using a file on
> your filesystem via loopback instead of messing about with a silly
> 1-cylinder partition.

I tried binding a loopback device to an md array, but it failed. I
(erroneously) concluded that loopback devices could not be used in md
arrays. Now, taking your advice, I tried again, and it worked. There's
a glitch, however: initially I tried on a 62 sector file, now on one
of the 64k files i extracted from the sda5+sda1 linear array I built
previously. If I revert to the smaller 62 sector file which would
allow me to duplicate the partition table, building the array fails.
Now I understand that this is due to the fact that the first device
(incidentally a loopback device) is smaller than the 64K rounding
size. This means that the loopback device size is rounded down to
zero. Hence, the error from mdadm.

This means that in the virtual setup I cannot maintain the partition
table as it is in the physical setup. That said, I agree that the sda5
approach is clumsy. Yet, even if I go through the loopback device I
will need to prepend a 64K device at the head of the windows
partition. And the virtual partition table will not coincide with the
physical one.

>> Thank you in advance for any help you can provide.
>
> Good luck and let me/us know how you get on!

Ok. Did it. I eventually used the loopback device as you suggested,
but with a 128 sector size. Now I am able to mount my Windows
partition via the md0p1 device.

fsc:~# fdisk /dev/md0

The number of cylinders for this disk is set to 30224768.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): u
Changing display/entry units to sectors

Command (m for help): p

Disk /dev/md0: 123.8 GB, 123800649728 bytes
2 heads, 4 sectors/track, 30224768 cylinders, total 241798144 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xde317135

    Device Boot      Start         End      Blocks   Id  System
/dev/md0p1   *         128   241798143   120899008    7  HPFS/NTFS

Command (m for help): Command (m for help): ^C
fsc:~# mount
/dev/sda3 on / type ext4 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc
(rw,noexec,nosuid,nodev)
/dev/md0p1 on /mnt/windoze type ntfs (rw)
fsc:~#

Nice feat! Now I have to get a bootloader installed, and I should be set.

Thanks again.

Alex

Re: Linear mode: array size is less than sum of sizes

[Alessandro Baretta]

On Tue, Jan 5, 2010 at 9:13 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
> On 05/01/2010 20:06, John Robinson wrote:
>
> Try playing with dm instead of md - `man dmsetup` suggests it may be able to
> do a linear concatenation in single-sector amounts...


Too late... I did it with md. But thanks for the suggestion: yes, this
was the right way to do this.

Alex

Re: Linear mode: array size is less than sum of sizes

[Alessandro Baretta]

Hi again John.

On Tue, Jan 5, 2010 at 9:06 PM, John Robinson
<john.robinson@anonymous.org.uk> wrote:
>
> I don't really know how safe it is, but I don't like your 16002-sector
> thingy. If you're going to do this I can't help thinking you should
> duplicate the real partition table, showing the partition starting on sector
> 63, so you want 63 extra sectors, not 16002. You might do it using a file on
> your filesystem via loopback instead of messing about with a silly
> 1-cylinder partition.

You were right. No way to get the Windows bootloader to even try
starting up Windows unless the windows partition is exactly at the
safe offset from as it is in the original drive. Too bad that Windows
dies with a BSOD while booting... That's with qemu-kvm 0.11.0 and
Linux 2.6.30 on Debian Sneezy. Oh, well. I'll just try to avoid
booting into Windows...

Alex

Re: Linear mode: array size is less than sum of sizes

[John Robinson]

On 06/01/2010 22:14, Alessandro Baretta wrote:
> On Tue, Jan 5, 2010 at 9:06 PM, John Robinson
> <john.robinson@anonymous.org.uk> wrote:
>> [...] If you're going to do this I can't help thinking you should
>> duplicate the real partition table, showing the partition starting on sector
>> 63, so you want 63 extra sectors, not 16002.
[...]
> You were right. No way to get the Windows bootloader to even try
> starting up Windows unless the windows partition is exactly at the
> safe offset from as it is in the original drive.

I had a feeling it might; I think there's another copy of the partition 
description inside the partition somewhere, at least involved in the 
boot process. Not sure though. You may also have problems if you don't 
have the right drivers to match the virtual environment in your Windows 
startup.

Anyway, you could still have a go at concatenating two partitions with 
dmsetup.

Cheers,

John.