Re: IDE HPA

Re: IDE HPA

[Greg Felix]

Kernel list,

A while ago there was some discussion on the list regarding the
behavior of the kernel in regards to its unconditional disabling of
host protected areas of hard drives.  I ran into a problem this causes
with some RAID controllers.  I've been discussing the problem with
both the ata-raid mailing list and Oliver.  I feel we should copy the
kernel list because we don't think the current behavior is the
desirable one.

Below is some discussion Oliver and I have had about it.

> > Sorry for taking up your time. I saw your emails recently to the Linux
> > kernel mailing list concerning IDE host protected areas.  You were
> > asking why they are unconditionally disabled.  Did anyone ever give
> > you a good response to your question?
> >
> 
> Hi Greg,
> 
> Alan Cox answered, but he focussed entirely on the point that in his opinion,
> the main reason for using HPAs is something like backward-compatibility of
> the drive with old BIOSses that have problems with large disks.
> 
> But to be honest, I have never ever heard about that being a motivation to use
> an HPA. And as far as I know, that was not the reason for introducing an HPA
> anyway.
> 
> As far as I know, some HW vendors store some diagnostic tools in an HPA.
> 
> > I have found a bug where my BIOS is storing some RAID metadata near
> > the end of a disk.  The problem i run into is that the end of the disk
> > is 20MB off when Linux counts the HPA.
> >
> 
> So you are sure that your RAID controller uses an HPA to store the metadata? I
> am asking because some RAID controllers simply cut away a moderate region
> from the end of the disks and present the OS with a smaller disk, which is
> but a virtual one. In that case, no HPA is used. It is rather like the MD
> driver works.

My RAID controller isn't using an HPA to store metadata.  It's simply
recognizing that there is an HPA and reading its 63 sector backwards
offset starting at totalSectors-sizeOfHPA.

> But of course, the Linux kernel simply shows whether an HPA is used or not.

Right.  I get the output at bootup time.  It reads that the HPA is
20MB.  Which is exactly the size of how far off the metadata is in
Linux (once the HPA is disabled).

> > Have you heard of any kernel parameters that disable the HPA disabling?
> >
> 
> There is no runtime variable, the code is run unconditionally, unfortunately.

I've found where the code is and it'd be a simple hack to fix it and
recompile, but I'm concerned that other people will run into this at
some point.  I think we or the people who make decisions ought to
revisit the disabling of HPAs idea.

> > Thanks for your time,
> > Greg Felix
> 
> Not at all! Should we CC the mail the kernel mailing list?

I think we should.  In fact, I will with this email.

> Best regards
> 
> Oliver

Re: IDE HPA

[Bartlomiej Zolnierkiewicz]

Hi,

OK, it seems, there is enough need for bringing back more control over HPA.

HPA shouldn't be disabled by default and new kernel parameter ("hdx=hpa")
should be added for disabling HPA (yep, people with buggy BIOS-es will
have to add this parameter to their kernel command line, sorry).

If somebody wants to go ahead and submit actual patches...
[s]he is welcomed to.

Thanks,
Bartlomiej

On 8/30/05, Greg Felix <greg.felix@gmail.com> wrote:
> Kernel list,
> 
> A while ago there was some discussion on the list regarding the
> behavior of the kernel in regards to its unconditional disabling of
> host protected areas of hard drives.  I ran into a problem this causes
> with some RAID controllers.  I've been discussing the problem with
> both the ata-raid mailing list and Oliver.  I feel we should copy the
> kernel list because we don't think the current behavior is the
> desirable one.
> 
> Below is some discussion Oliver and I have had about it.
> 
> > > Sorry for taking up your time. I saw your emails recently to the Linux
> > > kernel mailing list concerning IDE host protected areas.  You were
> > > asking why they are unconditionally disabled.  Did anyone ever give
> > > you a good response to your question?
> > >
> >
> > Hi Greg,
> >
> > Alan Cox answered, but he focussed entirely on the point that in his opinion,
> > the main reason for using HPAs is something like backward-compatibility of
> > the drive with old BIOSses that have problems with large disks.
> >
> > But to be honest, I have never ever heard about that being a motivation to use
> > an HPA. And as far as I know, that was not the reason for introducing an HPA
> > anyway.
> >
> > As far as I know, some HW vendors store some diagnostic tools in an HPA.
> >
> > > I have found a bug where my BIOS is storing some RAID metadata near
> > > the end of a disk.  The problem i run into is that the end of the disk
> > > is 20MB off when Linux counts the HPA.
> > >
> >
> > So you are sure that your RAID controller uses an HPA to store the metadata? I
> > am asking because some RAID controllers simply cut away a moderate region
> > from the end of the disks and present the OS with a smaller disk, which is
> > but a virtual one. In that case, no HPA is used. It is rather like the MD
> > driver works.
> 
> My RAID controller isn't using an HPA to store metadata.  It's simply
> recognizing that there is an HPA and reading its 63 sector backwards
> offset starting at totalSectors-sizeOfHPA.
> 
> > But of course, the Linux kernel simply shows whether an HPA is used or not.
> 
> Right.  I get the output at bootup time.  It reads that the HPA is
> 20MB.  Which is exactly the size of how far off the metadata is in
> Linux (once the HPA is disabled).
> 
> > > Have you heard of any kernel parameters that disable the HPA disabling?
> > >
> >
> > There is no runtime variable, the code is run unconditionally, unfortunately.
> 
> I've found where the code is and it'd be a simple hack to fix it and
> recompile, but I'm concerned that other people will run into this at
> some point.  I think we or the people who make decisions ought to
> revisit the disabling of HPAs idea.
> 
> > > Thanks for your time,
> > > Greg Felix
> >
> > Not at all! Should we CC the mail the kernel mailing list?
> 
> I think we should.  In fact, I will with this email.
> 
> > Best regards
> >
> > Oliver

Re: IDE HPA

[Alan Cox]

On Maw, 2005-08-30 at 09:52 -0600, Greg Felix wrote:
> Right.  I get the output at bootup time.  It reads that the HPA is
> 20MB.  Which is exactly the size of how far off the metadata is in
> Linux (once the HPA is disabled).

So your actual problem is nothing to do with the kernel or with the HPA
behaviour ? Whatever tool you are using for raid set up isn't reading
and processing the right fields. 

It isnt the kernels fault if you compute from of end of disk rather than
from end of non reserved area is it ?

Re: IDE HPA

[Alan Cox]

On Maw, 2005-08-30 at 18:16 +0200, Bartlomiej Zolnierkiewicz wrote:
> HPA shouldn't be disabled by default and new kernel parameter ("hdx=hpa")
> should be added for disabling HPA (yep, people with buggy BIOS-es will
> have to add this parameter to their kernel command line, sorry).

Thats large numbers of systems. Large numbers of disks as strapped for
32GB and other clipping arrangements. With a vendor hat on thats
unworkable because

a) It will stop thousands of people installing their systems
b) Many users will get horrible corruption when they update the kernel
and their box explodes as the fs tries to write to areas of disk that
have vanished mysteriously.

(and we know all about this because ancient kernels had options for
doing this in the compile that burned people)

So its a very bad idea indeed. A boot option for not disabling the hpa
is possibly sensible for a few users who want that, or simply getting
them to fix their buggy user space app would be even simpler.

The only way I can see to truely automate it for most cases would be to
snoop the partition table if its MSDOS format and see if the table
matches the HPA clipped disk or the non-HPA clipped disk. If it matches
the HPA clipped disk then you know not to fiddle. Otherwise its either a
new disk, clipped by the 32GB jumper, non-x86 disk etc in which case you
might as well disable any HPA.

Alan

Re: IDE HPA

[Bartlomiej Zolnierkiewicz]

On 8/30/05, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> On Maw, 2005-08-30 at 18:16 +0200, Bartlomiej Zolnierkiewicz wrote:
> > HPA shouldn't be disabled by default and new kernel parameter ("hdx=hpa")
> > should be added for disabling HPA (yep, people with buggy BIOS-es will
> > have to add this parameter to their kernel command line, sorry).
> 
> Thats large numbers of systems. Large numbers of disks as strapped for
> 32GB and other clipping arrangements. With a vendor hat on thats
> unworkable because
> 
> a) It will stop thousands of people installing their systems
> b) Many users will get horrible corruption when they update the kernel
> and their box explodes as the fs tries to write to areas of disk that
> have vanished mysteriously.
> 
> (and we know all about this because ancient kernels had options for
> doing this in the compile that burned people)
> 
> So its a very bad idea indeed. A boot option for not disabling the hpa
> is possibly sensible for a few users who want that, or simply getting
> them to fix their buggy user space app would be even simpler.

OK, boot option for disabling HPA for users that want it is a indeed most
sensible approach.

> The only way I can see to truely automate it for most cases would be to
> snoop the partition table if its MSDOS format and see if the table
> matches the HPA clipped disk or the non-HPA clipped disk. If it matches
> the HPA clipped disk then you know not to fiddle. Otherwise its either a
> new disk, clipped by the 32GB jumper, non-x86 disk etc in which case you
> might as well disable any HPA.
> 
> Alan

Re: IDE HPA

[Molle Bestefich]

Alan Cox wrote:
> Greg Felix wrote:
> > Right.  I get the output at bootup time.  It reads that the HPA is
> > 20MB.  Which is exactly the size of how far off the metadata is in
> > Linux (once the HPA is disabled).
> 
> So your actual problem is nothing to do with the kernel or with the HPA
> behaviour ? Whatever tool you are using for raid set up isn't reading
> and processing the right fields.

If the formula is to fix all the userspace apps to take into account a
potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
be fixed.  Because if you create a partition spanning your entire
disk, including the HPA area, and your boot files by some coincidence
ends up in the HPA part of the disk, the BIOS won't be able to read
them, and your system will not boot.  And if you fix those tools now,
what about users that use older Linux distributions?  They'll have a
parade of problems coming to them with their new HPA-enabled disks
because every userspace tool assumes that <BIOS sector count == Linux
sector count>.

> It isnt the kernels fault if you compute from of end of disk rather than
> from end of non reserved area is it ?

I agree that the entire disk should be visible under Linux.  But
instead of fixing every userspace app that assumes <BIOS sector count
== Linux sector count> (I'm guessing that's a lot), how about simply
exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
'disc' device ?

Re: IDE HPA

[Alan Cox]

> If the formula is to fix all the userspace apps to take into account a
> potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
> be fixed.  Because if you create a partition spanning your entire
> disk, including the HPA area, and your boot files by some coincidence
> ends up in the HPA part of the disk, the BIOS won't be able to read
> them, and your system will not boot.  And if you fix those tools now,

The distribution vendors already put the boot area low down the disk to
handle a whole variety of problems like this and the 1024 cylinder limit
on old BIOSes. 

> what about users that use older Linux distributions?  They'll have a
> parade of problems coming to them with their new HPA-enabled disks
> because every userspace tool assumes that <BIOS sector count == Linux
> sector count>.

I work for a vendor. I get most of our IDE layer bugs. And do you know
HPA doesn't really feature. The broken locking does, the hangs on
changing down speed by CRC do, numerous fascinating platform specific
bugs do, people with cables backwards do. HPA doesn't. It just works out
for people.

> > It isnt the kernels fault if you compute from of end of disk rather than
> > from end of non reserved area is it ?
> 
> I agree that the entire disk should be visible under Linux.  But
> instead of fixing every userspace app that assumes <BIOS sector count
> == Linux sector count> (I'm guessing that's a lot), how about simply
> exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
> 'disc' device ?


A lot of applications assume Linux reported size == real disk size. It
also wouldn't solve the case of a file system that spans both inside and
outside the HPA.

Alan

Re: IDE HPA

[Molle Bestefich]

Alan Cox wrote:
> > If the formula is to fix all the userspace apps to take into account a
> > potential HPA, then eg. FDISK + SFDISK + Disk Druid et al should also
> > be fixed.  Because if you create a partition spanning your entire
> > disk, including the HPA area, and your boot files by some coincidence
> > ends up in the HPA part of the disk, the BIOS won't be able to read
> > them, and your system will not boot.  And if you fix those tools now,
> 
> The distribution vendors already put the boot area low down the disk to
> handle a whole variety of problems like this and the 1024 cylinder limit
> on old BIOSes.

Only as an option - it's not always possible.
For example one might have a Windows partition already in place, etc..

> > what about users that use older Linux distributions?  They'll have a
> > parade of problems coming to them with their new HPA-enabled disks
> > because every userspace tool assumes that <BIOS sector count == Linux
> > sector count>.
> 
> I work for a vendor. I get most of our IDE layer bugs. And do you know
> HPA doesn't really feature.

Fair enough.  It took 5 seconds of googling to find a guy whose GRUB
stopped working because of the HPA, so at least *some* people are
already having problems...

> > > It isnt the kernels fault if you compute from of end of disk rather than
> > > from end of non reserved area is it ?
> >
> > I agree that the entire disk should be visible under Linux.  But
> > instead of fixing every userspace app that assumes <BIOS sector count
> > == Linux sector count> (I'm guessing that's a lot), how about simply
> > exporting the HPA as /dev/ide/hostX/busY/targetZ/lunA/hpa, next to the
> > 'disc' device ?
> 
> A lot of applications assume Linux reported size == real disk size.

I can't think of any, but you might be right.

> It also wouldn't solve the case of a file system that spans both inside and
> outside the HPA.

If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
create such a filesystem. I'm guessing it's not possible with Windows
either, or with any BIOS-based OS.

Creating filesystems that include the HPA defeats the entire idea of
the HPA in the first place.
  
If one does not care to use the HPA, one should disable it in the BIOS
entirely, so that everywhere (!) the entire disk is seen.

The inclusion of the HPA in /dev/.../disc should be a special case for
those unfortunate souls with laptops where the HPA cannot be disabled
because their BIOS is crap, IMHO.

Re: IDE HPA

[Matthew Garrett]

Molle Bestefich <molle.bestefich@gmail.com> wrote:

> If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> create such a filesystem. I'm guessing it's not possible with Windows
> either, or with any BIOS-based OS.

Such filesystems already exist. Changing this behaviour now would break
existing setups.

-- 
Matthew Garrett | mjg59-chiark.mail.linux-rutgers.kernel@srcf.ucam.org

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 15:33 +0200, Molle Bestefich wrote:
> > It also wouldn't solve the case of a file system that spans both inside and
> > outside the HPA.
> 
> If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> create such a filesystem. I'm guessing it's not possible with Windows
> either, or with any BIOS-based OS.

It is on many windows systems which include extra drivers that come with
large disks.

> Creating filesystems that include the HPA defeats the entire idea of
> the HPA in the first place.

See previous discussion, this is untrue. It may not have been the intent
of the standards authors but its certainly the idea of a large number of
disk and system vendors that the HPA is a handy way to deal with old
BIOSes and other items of that nature.
  
> If one does not care to use the HPA, one should disable it in the BIOS
> entirely, so that everywhere (!) the entire disk is seen.

And in the real world BIOSes don't get updated often by vendors let
alone by users.

Re: IDE HPA

[Molle Bestefich]

Alan Cox wrote:
> > If one does not care to use the HPA, one should disable it in the
> > BIOS entirely, so that everywhere (!) the entire disk is seen.
>
> And in the real world BIOSes don't get updated often
> by vendors let alone by users.

I meant the BIOS setup screen, not a firmware update...
Supposedly the BIOS can change the bounds of the HPA with special ATA commands..

Matthew Garrett wrote:
> Molle Bestefich wrote:
> > If HPA were exposed as /dev/.../hpa then it wouldn't be possible to
> > create such a filesystem. I'm guessing it's not possible with Windows
> > either, or with any BIOS-based OS.
> 
> Such filesystems already exist. Changing this behaviour now would break
> existing setups.

Not if, as proposed, there was a kernel switch to enable including the
HPA in the disc area.
Also those who has such filesystems {c,sh}ould disable HPA in their
BIOS (hopefully).

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 18:24 +0200, Molle Bestefich wrote:
> I meant the BIOS setup screen, not a firmware update...
> Supposedly the BIOS can change the bounds of the HPA with special ATA commands..

I've yet to see a BIOS that exposed the functionality

> Not if, as proposed, there was a kernel switch to enable including the
> HPA in the disc area.

And users magically knew about it - thats why it has to default the
other way.

Re: IDE HPA

[Molle Bestefich]

Alan Cox wrote:
> Molle Bestefich wrote:
> > Not if, as proposed, there was a kernel switch to enable including the
> > HPA in the disc area.
> 
> And users magically knew about it - thats why it has to default the
> other way.

Ok, so just to reiterate..

The current default is causing grief because dmraid doesn't work, grub
doesn't work and other userspace apps probably breaks too.  Users have
to google and post to mailing lists just to get things to work (... if
they could, which would require eg. a kernel option, but anyway).

The other way round, users would have to google to find the kernel
option that claims the HPA area (if they felt the need to overwrite
the BIOS's backup area), but those that felt the need would then be
rewarded with eg. 10 GB extra disk space.  And if they didn't feel
like it, their userspace apps would still work just fine.

>From my POV it's hard to see why the current default is sensible.

(We'll probably just have to agree on disagreeing, unless you can enlighten me.)

Related matters:
If you decide to include the HPA in one of your filesystems, is there
not a big risk that the BIOS will overwrite something there?

Re: IDE HPA

[Vojtech Pavlik]

On Fri, Sep 02, 2005 at 06:05:12PM +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 18:24 +0200, Molle Bestefich wrote:
> > I meant the BIOS setup screen, not a firmware update...
> > Supposedly the BIOS can change the bounds of the HPA with special ATA commands..
> 
> I've yet to see a BIOS that exposed the functionality

ThinkPads do let the user enable/disable it.

-- 
Vojtech Pavlik
SuSE Labs, SuSE CR

Re: IDE HPA

[Matthew Garrett]

Molle Bestefich <molle.bestefich@gmail.com> wrote:

> The other way round, users would have to google to find the kernel
> option that claims the HPA area (if they felt the need to overwrite
> the BIOS's backup area), but those that felt the need would then be
> rewarded with eg. 10 GB extra disk space.  And if they didn't feel
> like it, their userspace apps would still work just fine.

Unless they already have a filesystem that stretches over the HPA, in
which case the change in default behaviour would result in kernel
upgrades breaking everything horribly. It may well be the case that the
current situation is undesirable, but just changing it back *will break*
things.

-- 
Matthew Garrett | mjg59-chiark.mail.linux-rutgers.kernel@srcf.ucam.org

Re: IDE HPA

[Peter Jones]

On Fri, 2005-09-02 at 19:44 +0200, Molle Bestefich wrote:

> Related matters:
> If you decide to include the HPA in one of your filesystems, is there
> not a big risk that the BIOS will overwrite something there?

Isn't the bigger risk that if you include the HPA in your block device,
you'll overwrite your BIOS there?

At least that's what happens on some thinkpads.

It'd be nice if we left it as the BIOS leaves it by default, but make a
straightforward way to enable/disable/alter the HPA region from
software.

(if there's already a straightforward way, feel free to clue me in --
but the default should almost certainly be to assume the HPA is set up
correctly, shouldn't it?)
-- 
  Peter

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 19:44 +0200, Molle Bestefich wrote:
> The current default is causing grief because dmraid doesn't work, grub
> doesn't work and other userspace apps probably breaks too.  Users have
> to google and post to mailing lists just to get things to work (... if
> they could, which would require eg. a kernel option, but anyway).

You've show what - one obscure case with grub of an issue installers
already know about. dmraid is experimental anyway so submit fixes.

> The other way round, users would have to google to find the kernel
> option that claims the HPA area

And to find why their fs just choked, their computer no longer runs and
their files are lost. At which point they may not even be able to use
google.

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 14:09 -0400, Peter Jones wrote:
> (if there's already a straightforward way, feel free to clue me in --
> but the default should almost certainly be to assume the HPA is set up
> correctly, shouldn't it?)

The normal use of HPA is to clip drives to get them past BIOS boot
checks. The thinkpads come with a pre-installed partition table which
will protect the HPA unless the user goes to town removing it.

The ideal case would be that the partition table is considered at boot
to see if the HPA matches the partitiont table or not. You'd also then
need dynamic HPA enable/disable for installers and other tools to go
with that.

Send patches.

Re: IDE HPA

[Peter Jones]

On Fri, 2005-09-02 at 19:59 +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 14:09 -0400, Peter Jones wrote:
> > (if there's already a straightforward way, feel free to clue me in --
> > but the default should almost certainly be to assume the HPA is set up
> > correctly, shouldn't it?)
> 
> The normal use of HPA is to clip drives to get them past BIOS boot
> checks.

Ugh.  So some BIOSes use it for legitimate reasons (like thinkpads), and
some use it to work around BIOS bugs.  Great.

>  The thinkpads come with a pre-installed partition table which
> will protect the HPA unless the user goes to town removing it.

Mine didn't, but it does have an HPA.  Thankfully we weren't disabling
it yet when I installed my laptop -- I know others who weren't so lucky.
So this partitioning scheme hasn't always been the case...

And it seems broken anyway.  The point of the HPA is to make the OS see
a smaller/different disk layout, unless it's actually trying to update
things that are "protected", right?  If so, the partition table pointing
outside of the the disk when the HPA configuration hasn't been changed
from the bootup default totally broken :/

It really sounds better (to my naive mind, at least) to whitelist the
known-broken BIOSes.

> The ideal case would be that the partition table is considered at boot
> to see if the HPA matches the partitiont table or not. You'd also then
> need dynamic HPA enable/disable for installers and other tools to go
> with that.

Well, installers probably should be aware, yes -- that's why I mentioned
userland interfaces to enabling/disabling.  But to me it still seems
like we want to disable the HPA during installation and bootup, but only
if your BIOS is doing things wrong.

> Send patches.

Point taken.
-- 
  Peter

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 15:14 -0400, Peter Jones wrote:
> Ugh.  So some BIOSes use it for legitimate reasons (like thinkpads), and
> some use it to work around BIOS bugs.  Great.

All are legitimate uses. The partition table tells you which.

> Mine didn't, but it does have an HPA.  Thankfully we weren't disabling
> it yet when I installed my laptop -- I know others who weren't so lucky.
> So this partitioning scheme hasn't always been the case...

You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
This behaviour goes back pretty much to the creation of the ATA spec for
HPA. In fact if it was that long ago IBM shipped it with Windows so it
did have a partition table!

> It really sounds better (to my naive mind, at least) to whitelist the
> known-broken BIOSes.

Not really practical. You'd have to list most older PC systems.

> Well, installers probably should be aware, yes -- that's why I mentioned
> userland interfaces to enabling/disabling.  But to me it still seems
> like we want to disable the HPA during installation and bootup, but only
> if your BIOS is doing things wrong.

"Wrong" is a poor term here.

If the system has a partition table that doesn't cover the post HPA area
and its about the right size we can be fairly sure the right choice is
to honour the HPA, if its a randomly different size its a fair bet the
disk got moved

If the partition table exceeds the no HPA area of disk but not the full
disk then its almost certainly right the HPA should be disabled post
boot. If it exceeds both its a raid 0 volume of some form...

Re: IDE HPA

[Peter Jones]

On Fri, 2005-09-02 at 21:22 +0100, Alan Cox wrote:
> On Gwe, 2005-09-02 at 15:14 -0400, Peter Jones wrote:
> > Ugh.  So some BIOSes use it for legitimate reasons (like thinkpads), and
> > some use it to work around BIOS bugs.  Great.
> 
> All are legitimate uses. The partition table tells you which.
> 
> > Mine didn't, but it does have an HPA.  Thankfully we weren't disabling
> > it yet when I installed my laptop -- I know others who weren't so lucky.
> > So this partitioning scheme hasn't always been the case...
> 
> You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.

You may be right -- it's likely that I shrank my windows partition on
some other OS or Distro that wasn't designed with to screw up the disk.

> This behaviour goes back pretty much to the creation of the ATA spec for
> HPA. In fact if it was that long ago IBM shipped it with Windows so it
> did have a partition table!

Yes, it did have a partition table -- but the partition table did (and
still does) not include partitions which overlap the HPA.  Right now it
still appears as unused space.

> > It really sounds better (to my naive mind, at least) to whitelist the
> > known-broken BIOSes.
> 
> Not really practical. You'd have to list most older PC systems.

Most older PC systems use HPA?  Really?

The alternatives seem to be:

a) whitelist systems (in the kernel or userland) which use it the
   opposite way, where the space is being used for something
   relatively important (at least conceptually), or
b) bad heuristics to figure out which is which.

Both of these suck.  Have I missed something?

> > Well, installers probably should be aware, yes -- that's why I mentioned
> > userland interfaces to enabling/disabling.  But to me it still seems
> > like we want to disable the HPA during installation and bootup, but only
> > if your BIOS is doing things wrong.
> 
> "Wrong" is a poor term here.
> 
> If the system has a partition table that doesn't cover the post HPA area
> and its about the right size we can be fairly sure the right choice is
> to honour the HPA, if its a randomly different size its a fair bet the
> disk got moved
> 
> If the partition table exceeds the no HPA area of disk but not the full
> disk then its almost certainly right the HPA should be disabled post
> boot. If it exceeds both its a raid 0 volume of some form...

So where would you envision this code to check the partition table, the
HPA/host default disk size, and guess how things should be set up?

>From a userland perspective, it's very difficult to let users know
they'll be screwing themselves by partitioning the entire disk, so we
really should be leaving HPA enabled if the protected area is indeed not
for consumption.

Also, the heuristic is harder than this -- if we reexamine the fakeraid
case, then it's clear we have to look for raid metadata, figure out if
the raid includes stuff inside the HPA or not, and then if it doesn't we
don't care -- but that's assuming there _is_ raid metadata.

Long term, many people hope, possibly unrealistically, that we'll be
able to write out raid metadata for people creating raids on cards which
support fakeraid, and have the BIOS grok it appropriately.  So in that
case, we may well have a blank (or garbage) disk, and we can't check the
partition table or any raid metadata.  Correct me if I'm wrong, but I
don't see a simple heuristic for this case.

(as a side note, I know one user who, at OLS, noticed we fail to
re-initialize HPA after unsuspend, so on at least t40 the disk gets
smaller when you suspend.  This may or may not be fixed, I haven't
checked.  But it's one more sort of pain we get into by disabling it,
whether justified or not.)

I think if we go the heuristic route, then the *safest* option is to
leave it enabled by default and let userland installers/initrd do fixups
by telling the kernel to change the state.  That way it can keep it
correct on suspend/unsuspend, but we don't give users the opportunity to
really screw themselves unless they try pretty hard, and we don't
actually screw the users unless we simply get the heuristic totally
wrong.  It's also consistent with the more general policy of leaving
policy up to the userland.

But your point about how I should send you a patch certainly still
applies.

-- 
  Peter

Re: IDE HPA

[Pekka Pietikainen]

On Fri, Sep 02, 2005 at 09:22:58PM +0100, Alan Cox wrote:
> You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
> This behaviour goes back pretty much to the creation of the ATA spec for
> HPA. In fact if it was that long ago IBM shipped it with Windows so it
> did have a partition table!
FC3 happily ignored the HPA on IBM X31. FC4 did not. I won't vow about the
original partitioning, but a "worked for just about everything" FC3
kickstart slightly updated to FC4 started breaking horribly after suspend.
As in messed up filesystems since parts of the disk just vanished when you
resumed. (FC3 could have been broken too, but CONFIG_IDE_STROKE or whatnot
wasn't enabled, so it worked as expected). It probably didn't work
as expected for people with broken bioses that didn't do >32GB ether, but those
people required additional hacks for competing OS's too, so it wasn't such a
biggie for them, I suppose.

Some sort of BIOS bug, completely IBM's (or rather some subcontractor)
fault, happens on one X31 laptop I know of, where the HPA just can't be
disabled. At all. The BIOS setting gets ignored. On the one I personally use
disabling it works, so losing the recovery Windows XP was enough to have a
functional system. Not optimal, but I don't really need the recovery stuff
for anything, so might as well use the entire disk.

For the one where disabling the HPA just didn't work the solution was to
manually partition, and just not using the area the HPA would appear on.
There goes automatic kickstart installs, but at least the laptop now is usable.

Re: IDE HPA

[Alan Cox]

On Gwe, 2005-09-02 at 17:14 -0400, Peter Jones wrote:
> > You installed it on Red Hat 7 ? I think 7, may have been 6.x or earlier.
> 
> You may be right -- it's likely that I shrank my windows partition on
> some other OS or Distro that wasn't designed with to screw up the disk.

If you shrink existing partitions it won't ever screw you up. The
geometry data for the partition table spans only the non HPA area.

> Yes, it did have a partition table -- but the partition table did (and
> still does) not include partitions which overlap the HPA.  Right now it
> still appears as unused space.

But they are also on the IBM I looked at are obvious because the
geometry in the partition table does not span the HPA so the problem
doesn't arise as confusion.

> > Not really practical. You'd have to list most older PC systems.
> 
> Most older PC systems use HPA?  Really?

Many of those "magic windows drive/bios fixup" type programs work by
having the jumper on the drive set the HPA and the drive report a
smaller size, then the windows magic driver undoes this.


> Both of these suck.  Have I missed something?


I fear not.

> So where would you envision this code to check the partition table, the
> HPA/host default disk size, and guess how things should be set up?

fdisk and friends already have to parse and process the existing
partition tables.

> they'll be screwing themselves by partitioning the entire disk, so we
> really should be leaving HPA enabled if the protected area is indeed not
> for consumption.

Define "not for consumption". It should be *hard* to use it, and it
should not occur by accident. Deliberately is a different matter. And
that should be a run time not boot time action.

> (as a side note, I know one user who, at OLS, noticed we fail to
> re-initialize HPA after unsuspend, so on at least t40 the disk gets
> smaller when you suspend.  This may or may not be fixed, I haven't
> checked.  But it's one more sort of pain we get into by disabling it,
> whether justified or not.)

Known problem. ACPI provides the correct infrastructure for much of this
but the IDE layer doesn't support it. Send patches to Bartlomiej. The
core infrastructure is there because Andre saw the need for the ACPI
taskfile support coming. The HPA restore is just another step in the
state machine for resume and quite doable. Good little project for
someone wanting to play with the IDE layer.

> I think if we go the heuristic route, then the *safest* option is to
> leave it enabled by default and let userland installers/initrd do fixups
> by telling the kernel to change the state. 

Assuming we are talking about hda1/2/... then the partitions are already
clipped by the OS to the volume size. We could conceivably make the size
of the disk itself writable. We don't need to get into programming drive
HPA when we can do it ourselves, and we can clip non HPA capable drives
too should someone find a cause for it.

Re: IDE HPA

[Jeff Garzik]

Peter Jones wrote:
> So where would you envision this code to check the partition table, the
> HPA/host default disk size, and guess how things should be set up?
> 
>>From a userland perspective, it's very difficult to let users know
> they'll be screwing themselves by partitioning the entire disk, so we
> really should be leaving HPA enabled if the protected area is indeed not
> for consumption.
> 
> Also, the heuristic is harder than this -- if we reexamine the fakeraid
> case, then it's clear we have to look for raid metadata, figure out if
> the raid includes stuff inside the HPA or not, and then if it doesn't we
> don't care -- but that's assuming there _is_ raid metadata.
> 
> Long term, many people hope, possibly unrealistically, that we'll be
> able to write out raid metadata for people creating raids on cards which
> support fakeraid, and have the BIOS grok it appropriately.  So in that
> case, we may well have a blank (or garbage) disk, and we can't check the
> partition table or any raid metadata.  Correct me if I'm wrong, but I
> don't see a simple heuristic for this case.
> 
> (as a side note, I know one user who, at OLS, noticed we fail to
> re-initialize HPA after unsuspend, so on at least t40 the disk gets
> smaller when you suspend.  This may or may not be fixed, I haven't
> checked.  But it's one more sort of pain we get into by disabling it,
> whether justified or not.)

It seems to me that one should write an ATA-specific Device Mapper 
driver, which layers on top of an ATA disk.  The driver obtains the 
starting location of HPA, then exports two block devices:  one for the 
primary data area, and one for the HPA.

For situations where we want the start Linux philosophy -- Linux exports 
100% of the hardware capability -- no DM layer needs to be used.  For 
situations where its better to treat the HPA as a separate and distinct 
area, the DM driver would come in handy.

This follows the same philosophy as fakeraid (BIOS RAID):  we simply 
export the entire disk, and Device Mapper (google for 'dmraid') handles 
the vendor-proprietary RAID metadata.

	Jeff

Re: IDE HPA

[Bill Davidsen]

Jeff Garzik wrote:

> It seems to me that one should write an ATA-specific Device Mapper 
> driver, which layers on top of an ATA disk.  The driver obtains the 
> starting location of HPA, then exports two block devices:  one for the 
> primary data area, and one for the HPA.

I've stayed out of this, but that sounds like a perfect solution to move 
the choice back to the user. However, installers still need to be aware 
of it at initial Linux install, and give the user some rational options:
   - ignore it
   - leave alone but visible
   - blow it away and use the whole drive

It feels as if that's where the future disposition needs to be made. I 
do like treating the HPA as a separate drive though.
> 
> For situations where we want the start Linux philosophy -- Linux exports 
> 100% of the hardware capability -- no DM layer needs to be used.  For 
> situations where its better to treat the HPA as a separate and distinct 
> area, the DM driver would come in handy.
> 
> This follows the same philosophy as fakeraid (BIOS RAID):  we simply 
> export the entire disk, and Device Mapper (google for 'dmraid') handles 
> the vendor-proprietary RAID metadata.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me