support for drives larger than 2TiB

support for drives larger than 2TiB

[Tejun Heo]

Hello,

I've been playing with a SATA 2.5T drive and things don't look too
bad.  All four controllers I've tested worked fine and the driver and
kernel worked just fine.  Even BIOSes don't seem too bad.  At least
the two boards I tested (both about three years old) didn't have much
problem recognizing upto 2TiB and could access and boot fine although
I'm fairly sure there will be BIOSes which would behave erratically.

I also tested installing w/ openSUSE 11.3 and it worked fine.  It
automatically chose GPT and alignment and everything just worked (tm).
I think the situation shouldn't be too different for any distro which
uses up-to-date parted.

The only problem is that everything which is necessary for booting
needs to be located below 2TiB limit.  Please note that this is much
stricter restriction than the 128GiB limit we had due to LBA28.  That
limit was caused by BIOSes using LBA28 and vendors could and did
update and be done with it in many cases.  However, 2TiB limit is
inherent in the BIOS programming interface and currently the only way
to overcome it is using a completely different BIOS interface (EFI,
that is).  Vendors are not likely to introduce EFI for already
released products although they're much more likely to release updates
so that BIOSes can access upto 2TiB if they don't work already.  We'll
be stuck with 2TiB limit on much more configurations for longer period
of time.

So, distro installers need to try to locate everything needed for
bootstrapping below 2TiB limit (ie. a dedicated boot partition below
the limit).  Drives > 2TiB aren't on the market yet but aren't too far
away.  Let's make sure things will be ready by the next distro release
cycle.

Thanks.

-- 
tejun

Re: support for drives larger than 2TiB

[Henrique de Moraes Holschuh]

Message forwarded to debian-boot@lists.debian.org, and also to the Debian
BTS, package "debian-installer", bug #590169.

Thanks for the analysis and advice, Tejun.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: support for drives larger than 2TiB

[Alex Buell]

On Sat, 2010-07-24 at 11:58 +0200, Tejun Heo wrote:
> The only problem is that everything which is necessary for booting
> needs to be located below 2TiB limit.  Please note that this is much
> stricter restriction than the 128GiB limit we had due to LBA28.  That
> limit was caused by BIOSes using LBA28 and vendors could and did
> update and be done with it in many cases.  However, 2TiB limit is
> inherent in the BIOS programming interface and currently the only way
> to overcome it is using a completely different BIOS interface (EFI,
> that is).  Vendors are not likely to introduce EFI for already
> released products although they're much more likely to release updates
> so that BIOSes can access upto 2TiB if they don't work already.  We'll
> be stuck with 2TiB limit on much more configurations for longer period
> of time. 

The only thing that would please me no end with newer replacements for
BIOS is the ability to have 4k boot sectors. Imagine what we can do with
4k what we can't do with 512 bytes. 
-- 
http://www.munted.org.uk

One very high maintenance cat living here.

Re: support for drives larger than 2TiB

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Sat, 24 Jul 2010 13:36:06 BST, Alex Buell said:

> The only thing that would please me no end with newer replacements for
> BIOS is the ability to have 4k boot sectors. Imagine what we can do with
> 4k what we can't do with 512 bytes. 

Are you saying that 4K sectors have some special nice implications for the boot
process, or that the boot process is the last major hangup to fully supporting
a device with 4K sectors, which would give us an 8X boost in capacity on all
the codepaths that work via sector humbers?  I suspect you mean the latter, but
it's early in the morning  still.. ;)

As a side consideration - moving from 512 to 4K moves the associated limit from
2 TiB to 16 TiB.  Given the current rate of device density increase, how much
time will that buy us, and what do we do then?


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: support for drives larger than 2TiB

[Alex Buell]

On Sat, 2010-07-24 at 09:49 -0400, Valdis.Kletnieks@vt.edu wrote:
> On Sat, 24 Jul 2010 13:36:06 BST, Alex Buell said:
> 
> > The only thing that would please me no end with newer replacements for
> > BIOS is the ability to have 4k boot sectors. Imagine what we can do with
> > 4k what we can't do with 512 bytes. 
> 
> Are you saying that 4K sectors have some special nice implications for the boot
> process, or that the boot process is the last major hangup to fully supporting
> a device with 4K sectors, which would give us an 8X boost in capacity on all
> the codepaths that work via sector humbers?  I suspect you mean the latter, but
> it's early in the morning  still.. ;)

It would be interesting to see how newer BIOSses cope with 4k boot
sectors. I'm sure the latest ATAPI standards do allow 4k boot sectors, I
just want to know how this will be implemented for devices with larger
physical sectors than the more usual 512 byte sectors. 

> As a side consideration - moving from 512 to 4K moves the associated limit from
> 2 TiB to 16 TiB.  Given the current rate of device density increase, how much
> time will that buy us, and what do we do then?

There are now 3TB devices out there. But noone can boot from 4k devices
yet on existing PC systems. 

I'm sure a market to provide 3rd party BIOSes able to do this will
develop shortly. I know of one: coreboot. 
-- 
http://www.munted.org.uk

One very high maintenance cat living here.

Re: support for drives larger than 2TiB

[Stan Hoeppner]

Valdis.Kletnieks@vt.edu put forth on 7/24/2010 8:49 AM:

> As a side consideration - moving from 512 to 4K moves the associated limit from
> 2 TiB to 16 TiB.  Given the current rate of device density increase, how much
> time will that buy us, and what do we do then?

The crystal ball tells me that SSD adoption will kick this date a little
farther into the future given that SSD capacity lags mechanical by a large
margin.  In the not too distance future, PCs/laptops/netbooks will all
transition to shipping with SSD as their sole/main internal storage device,
with USB3/eSATA thumb drives of 1TB and above permanently replacing external
USB/eSATA mechanical drives.

Servers will probably transition at about the same time to shipping with
solely SSD storage internally for boot/OS and maybe some other primary and a
little secondary storage, with mechanical storage being optional, whether
internal or external.  Mechanical in the form of FC/iSCSI SAN and NAS arrays
will still rule large data needs well after the aforementioned transitions above.

-- 
Stan

Re: support for drives larger than 2TiB

[Tejun Heo]

Hello,

On 07/25/2010 01:08 AM, Stan Hoeppner wrote:
> Valdis.Kletnieks@vt.edu put forth on 7/24/2010 8:49 AM:
> 
>> As a side consideration - moving from 512 to 4K moves the associated limit from
>> 2 TiB to 16 TiB.  Given the current rate of device density increase, how much
>> time will that buy us, and what do we do then?

I don't really see it happening on ATA hard drives.  Maybe on USB /
external ones.  There simply are too many compatibility implications.

> In the not too distance future, PCs/laptops/netbooks will all
> transition to shipping with SSD as their sole/main internal storage
> device, with USB3/eSATA thumb drives of 1TB and above permanently
> replacing external USB/eSATA mechanical drives.

After hearing the same story for the good part of the decade, I'm a
bit skeptical about this too.  The unit capacity price ratio hasn't
really changed that much.  Maybe requirements for storage capacity
have reached / are approaching saturation point, so as long as the
price keeps coming down, SSDs can be mainstream without closing the
unit capacity price difference.  At this point, hybrid seems much more
realistic to me, but then again, maybe we really are reaching the
saturation point.  Who knows?

Thanks.

-- 
tejun

Re: support for drives larger than 2TiB

[Mikael Abrahamsson]

On Sun, 25 Jul 2010, Tejun Heo wrote:

> saturation point.  Who knows?

I don't think any design decision should be made on the assumption that 
drives won't get larger.

There is always a market for bigger capacity drives. Today I use SSD for 
system drive, and then much larger capacity drives for long term storage. 
The term "drives are either new or full" still holds true in the 2TB era, 
it'd be the same if there were 8TB drives available.

If it was economically possible to store uncompressed video I'm sure we 
would, if it was anywhere economical. With 1080p60 that's a LOT of data.

If there is cheap storage available, people will stop erasing data and 
just let it pile up. I know I do, but I'm less cost sensitive than most.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

RE: support for drives larger than 2TiB

[Yuhong Bao]

> However, 2TiB limit is
> inherent in the BIOS programming interface and currently the only way
> to overcome it is using a completely different BIOS interface (EFI,
> that is).
Nope, look at the Int13 extensions, it already support 64-bit LBA.

Yuhong Bao
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

RE: support for drives larger than 2TiB

[Yuhong Bao]

> However, 2TiB limit is
> inherent in the BIOS programming interface and currently the only way
> to overcome it is using a completely different BIOS interface (EFI,
> that is).
Nope, look at the Int13 extensions, it already support 64-bit LBA.

Yuhong Bao
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

Re: support for drives larger than 2TiB

[Greg Freemyer]

On Sat, Jul 24, 2010 at 2:40 PM, Yuhong Bao <yuhongbao_386@hotmail.com> wrote:
>
>> However, 2TiB limit is
>> inherent in the BIOS programming interface and currently the only way
>> to overcome it is using a completely different BIOS interface (EFI,
>> that is).
> Nope, look at the Int13 extensions, it already support 64-bit LBA.
>
> Yuhong Bao

64-bit LBA?  I assume you meant 48-bit LBA from ATA-7.  It will
address up to PBs I believe.  (KB, MB, GB, TB, PB, ....)

But I've seen lots of "48-bit LBA" supporting controllers that hit
limits at 500GB, 1TB, etc. and needed new firmware updates.

Secondly, if you look at the table on the right of
http://en.wikipedia.org/wiki/Master_boot_record#Disk_partitioning you
see that the starting sector of a partition is defined with a 32-bit
value.

ie. 2TB with 512 byte sectors.

The normal solution is to move to a GPT which requires EFI if you want
to boot from it.

Greg

Re: support for drives larger than 2TiB

[Henrique de Moraes Holschuh]

On Sat, 24 Jul 2010, Greg Freemyer wrote:
> Secondly, if you look at the table on the right of
> http://en.wikipedia.org/wiki/Master_boot_record#Disk_partitioning you
> see that the starting sector of a partition is defined with a 32-bit
> value.
> 
> ie. 2TB with 512 byte sectors.
> 
> The normal solution is to move to a GPT which requires EFI if you want
> to boot from it.

Can't one have a GPT with a guard boot sector which can read the real
partition table?

I think Tejun must mean some big problem at the BIOS APIs required for the
bootloader.  Tejun, what exactly croaks at the 2TB boundary (with 512KiB
sectors)?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: support for drives larger than 2TiB

[Tejun Heo]

Hello,

On 07/25/2010 01:22 AM, Henrique de Moraes Holschuh wrote:
> On Sat, 24 Jul 2010, Greg Freemyer wrote:
>> Secondly, if you look at the table on the right of
>> http://en.wikipedia.org/wiki/Master_boot_record#Disk_partitioning you
>> see that the starting sector of a partition is defined with a 32-bit
>> value.
>>
>> ie. 2TB with 512 byte sectors.
>>
>> The normal solution is to move to a GPT which requires EFI if you want
>> to boot from it.
> 
> Can't one have a GPT with a guard boot sector which can read the real
> partition table?

Yeah, parted already does that although there seem to be some bugs and
they sometimes are inconsistent.  In principle, BIOSes don't really
care about the partition table (they look at it primarily to figure
out geometry and stuff) and can boot as long as the initial boot
sector is accessible.

> I think Tejun must mean some big problem at the BIOS APIs required for the
> bootloader.  Tejun, what exactly croaks at the 2TB boundary (with 512KiB
> sectors)?
	
The 2TiB limit is coming from BIOS software interrupt for disk access
using 32bit for addressing.  If there's 64bit extension to BIOS disk
access, everything should work but still locating boot partition under
2TiB is pretty much required at this point, I think.

Thanks.

-- 
tejun

RE: support for drives larger than 2TiB

[Yuhong Bao]

> 64-bit LBA? I assume you meant 48-bit LBA from ATA-7. It will
> address up to PBs I believe. (KB, MB, GB, TB, PB, ....)
Yes, the sector number in Int13 extensions really is 64-bit.
Look at the Ralf Brown Interrupt List, for example.

Yuhong Bao
 		 	   		  
_________________________________________________________________
The New Busy is not the old busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3

RE: support for drives larger than 2TiB

[Yuhong Bao]

> 64-bit LBA? I assume you meant 48-bit LBA from ATA-7. It will
> address up to PBs I believe. (KB, MB, GB, TB, PB, ....)
Yes, the sector number in Int13 extensions really is 64-bit.
Look at the Ralf Brown Interrupt List, for example.

Yuhong Bao
 		 	   		  
_________________________________________________________________
The New Busy is not the old busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3

Re: support for drives larger than 2TiB

[Tejun Heo]

(cc'ing hpa, hi!)
Hello,

On 07/24/2010 08:40 PM, Yuhong Bao wrote:
>> However, 2TiB limit is
>> inherent in the BIOS programming interface and currently the only way
>> to overcome it is using a completely different BIOS interface (EFI,
>> that is).
>
> Nope, look at the Int13 extensions, it already support 64-bit LBA.

Ah, yeah, that sounds much more logical than requiring transition to
EFI.  Does anyone know how wide spread the support for this extension
is in BIOSes?  Do the bootloaders support this?

Thanks.

-- 
tejun

Re: support for drives larger than 2TiB

[H. Peter Anvin]

On 07/25/2010 01:01 AM, Tejun Heo wrote:
> (cc'ing hpa, hi!)
> Hello,
> 
> On 07/24/2010 08:40 PM, Yuhong Bao wrote:
>>> However, 2TiB limit is
>>> inherent in the BIOS programming interface and currently the only way
>>> to overcome it is using a completely different BIOS interface (EFI,
>>> that is).
>>
>> Nope, look at the Int13 extensions, it already support 64-bit LBA.
> 
> Ah, yeah, that sounds much more logical than requiring transition to
> EFI.  Does anyone know how wide spread the support for this extension
> is in BIOSes?  Do the bootloaders support this?
> 
> Thanks.
> 

Syslinux 4 supports it with full 64-bit capabilities.  SeaBIOS (used in
Qemu/KVM) supports it, and I know at least some hardware/firmware RAID
solutions support it; I recently got access to a 3 TB SATA drive but due
to NDA requirements I can't reveal the results of that testing.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: support for drives larger than 2TiB

[Tejun Heo]

Hello,

On 07/25/2010 10:07 AM, H. Peter Anvin wrote:
> Syslinux 4 supports it with full 64-bit capabilities.  SeaBIOS (used in
> Qemu/KVM) supports it, and I know at least some hardware/firmware RAID
> solutions support it; I recently got access to a 3 TB SATA drive but due
> to NDA requirements I can't reveal the results of that testing.

Hmmm... doesn't sound too bad then.  Is there an easy to test whether
64bit LBA works?  I have a 2.5TiB drive and can test the machines I
have and probably publish the result too.

Thanks.

-- 
tejun

Re: support for drives larger than 2TiB

[H. Peter Anvin]

On 07/25/2010 01:20 AM, Tejun Heo wrote:
> Hello,
> 
> On 07/25/2010 10:07 AM, H. Peter Anvin wrote:
>> Syslinux 4 supports it with full 64-bit capabilities.  SeaBIOS (used in
>> Qemu/KVM) supports it, and I know at least some hardware/firmware RAID
>> solutions support it; I recently got access to a 3 TB SATA drive but due
>> to NDA requirements I can't reveal the results of that testing.
> 
> Hmmm... doesn't sound too bad then.  Is there an easy to test whether
> 64bit LBA works?  I have a 2.5TiB drive and can test the machines I
> have and probably publish the result too.
> 

If you can set up the disk with GPT then just set up a bootable
partition beyond the 2 TiB mark and install Syslinux 4 on it.  (Note:
there is a bug in current versions of gdisk: attribute 2 is the legacy
BIOS bootable attribute, but the author got the bit order wrong and so
you have to set "attribute 61".  parted doesn't support the legacy BIOS
bootable attribute yet.)

If you don't want tomodify the contents of the disk then it can still be
tested, but I don't have a ready-made test for you.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: support for drives larger than 2TiB

[Tejun Heo]

Hello,

On 07/25/2010 08:52 PM, H. Peter Anvin wrote:
> If you can set up the disk with GPT then just set up a bootable
> partition beyond the 2 TiB mark and install Syslinux 4 on it.  (Note:
> there is a bug in current versions of gdisk: attribute 2 is the legacy
> BIOS bootable attribute, but the author got the bit order wrong and so
> you have to set "attribute 61".  parted doesn't support the legacy BIOS
> bootable attribute yet.)
>
> If you don't want tomodify the contents of the disk then it can still be
> tested, but I don't have a ready-made test for you.

The disk doesn't contain any data.  I'll try it with several different
motherboards and report back.

Thanks.

-- 
tejun

Re: support for drives larger than 2TiB

[Alex Buell]

On Sun, 2010-07-25 at 01:07 -0700, H. Peter Anvin wrote:
> 
> Syslinux 4 supports it with full 64-bit capabilities.  SeaBIOS (used
> in
> Qemu/KVM) supports it, and I know at least some hardware/firmware RAID
> solutions support it; I recently got access to a 3 TB SATA drive but
> due to NDA requirements I can't reveal the results of that testing. 

Yes but is it possible to boot from a complete 4k sector with the new
standards? 
-- 
http://www.munted.org.uk

One very high maintenance cat living here.

Re: support for drives larger than 2TiB

[H. Peter Anvin]

On 07/25/2010 01:26 AM, Alex Buell wrote:
> On Sun, 2010-07-25 at 01:07 -0700, H. Peter Anvin wrote:
>>
>> Syslinux 4 supports it with full 64-bit capabilities.  SeaBIOS (used
>> in
>> Qemu/KVM) supports it, and I know at least some hardware/firmware RAID
>> solutions support it; I recently got access to a 3 TB SATA drive but
>> due to NDA requirements I can't reveal the results of that testing. 
> 
> Yes but is it possible to boot from a complete 4k sector with the new
> standards? 

Yes, but currently there are no 4k *logical* sector products on the
market (and $DEITY knows how many BIOSes would handle them correctly.)
The BIOS interfaces should handle them fine, though.

	-hpa

-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

RE: support for drives larger than 2TiB

[Yuhong Bao]

> I've been playing with a SATA 2.5T drive and things don't look too
> bad. All four controllers I've tested worked fine and the driver and
> kernel worked just fine. Even BIOSes don't seem too bad. At least
> the two boards I tested (both about three years old) didn't have much
> problem recognizing upto 2TiB and could access and boot fine although
> I'm fairly sure there will be BIOSes which would behave erratically.

In fact, Seagate is already selling 3TB external drives and CrunchGear
was already able to get the internal 3TB drive inside out:
http://techcrunch.com/2010/07/01/hands-on-the-seagate-freeagent-goflex-desk-3tb-external-hard-drive/

Yuhong Bao
 		 	   		  
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1

RE: support for drives larger than 2TiB

[Yuhong Bao]

> I've been playing with a SATA 2.5T drive and things don't look too
> bad. All four controllers I've tested worked fine and the driver and
> kernel worked just fine. Even BIOSes don't seem too bad. At least
> the two boards I tested (both about three years old) didn't have much
> problem recognizing upto 2TiB and could access and boot fine although
> I'm fairly sure there will be BIOSes which would behave erratically.

In fact, Seagate is already selling 3TB external drives and CrunchGear
was already able to get the internal 3TB drive inside out:
http://techcrunch.com/2010/07/01/hands-on-the-seagate-freeagent-goflex-desk-3tb-external-hard-drive/

Yuhong Bao
 		 	   		  
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1

Re: support for drives larger than 2TiB

[Mark Lord]

On 10-07-24 02:48 PM, Yuhong Bao wrote:
>
> In fact, Seagate is already selling 3TB external drives and CrunchGear
> was already able to get the internal 3TB drive inside out:
> http://techcrunch.com/2010/07/01/hands-on-the-seagate-freeagent-goflex-desk-3tb-external-hard-drive/


The key word there is _external_ --> by using USB3/2/FW interfaces,
they neatly sidestep most BIOS issues.  For the time being anyway.

RE: support for drives larger than 2TiB

[Yuhong Bao]

> > In fact, Seagate is already selling 3TB external drives and CrunchGear
> > was already able to get the internal 3TB drive inside out:
> > http://techcrunch.com/2010/07/01/hands-on-the-seagate-freeagent-goflex-desk-3tb-external-hard-drive/
>
>
> The key word there is _external_ --> by using USB3/2/FW interfaces,
> they neatly sidestep most BIOS issues. For the time being anyway.
>
A more detailed review from AnandTech:
http://www.anandtech.com/show/3858/the-worlds-first-3tb-hdd-seagate-goflex-desk-3tb-review
 		 	   		  

RE: support for drives larger than 2TiB

[Yuhong Bao]

> > In fact, Seagate is already selling 3TB external drives and CrunchGear
> > was already able to get the internal 3TB drive inside out:
> > http://techcrunch.com/2010/07/01/hands-on-the-seagate-freeagent-goflex-desk-3tb-external-hard-drive/
>
>
> The key word there is _external_ --> by using USB3/2/FW interfaces,
> they neatly sidestep most BIOS issues. For the time being anyway.
>
A more detailed review from AnandTech:
http://www.anandtech.com/show/3858/the-worlds-first-3tb-hdd-seagate-goflex-desk-3tb-review