[RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Szakacsits Szabolcs]

          [kernel related notes are at the end]
Hello,

On 2 Jul 2004, Patrick J. LoPresti wrote:

Parted is looking for (co-)maintainers. Wouldn't you like to be?
Guillaume? Somebody from SUSE, Red Hat, Debian, anybody? I think 
it's a real challenge in its current state ;)

> Parted needs a mechanism to let me FORCE the geometry it uses.  Every
> other partitioning tool has this, usually via command-line switch.

FORCE or FIX if it's _asked_. Fix but not the way parted does now,
silently playing lottery. Only if user explicitely wants to do that, aka
user knows he has a problem, e.g. old parted messed up the partition
table.

The majority of users doesn't really know what is bootloader, MBR,
partition table, geometry, cylinder, sectors, etc and actually they
shouldn't even know.

Currently they blame the BIOS, LILO, GRUB, NTFS, FAT32, Microsoft,
bootloaders, kernel, hardware, firmware, themself(!!), etc. Parted 
is so hidden, embedded in tools, installers and behaves so randomly 
then I think it was very difficult to realise how broken it is, over
the years.

> Having such [geometry] guesswork in Parted itself is a design error,

Yes. Parted must get the geometry from the partition table unless

	1) the partition table is empty

	2) asked to fix the partition table

	3) asked to use user provided values

1) and 2) need a way to get a "sane" geometry from the BIOS or kernel.

Nevertheless, fixing Parted and all broken tools, that trust the kernel
getting the most-of-the-time-right-geometry, won't fix the problem
immediately because nobody can replace all these tools in the wild from
one day to the other. Transition would take several years.

Does anybody find the new HDIO_GETGEO semantic useful? Did it help
_anything_? 

Because the semantic change did break many people data, installations
permanently and keeps doing so every day.

Please also note, so far nobody stepped forward to fix parted.

	Szaka

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andries Brouwer]

On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:

> Please also note, so far nobody stepped forward to fix parted.

Nobody asked, but I wouldnt mind fixing this particular
aspect of parted.

If nobody else wants to maintain it I can take it, but then
"maintain" means: zero development, just bugfixes.

Andries

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andries Brouwer]

On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:

> Does anybody find the new HDIO_GETGEO semantic useful? Did it help
> _anything_? 

Yes. I do.

There was a steady stream of people reporting on geometry problems.
All these problems were caused by kernel guessing stuff.
If the kernel no longer volunteers a guess, then we no longer have
the situation that the guess can be wrong.

We lived in a world where things got more and more complicated
with programs guessing what values other programs might want
to satisfy a third program.

The new world is getting much simpler. Only the programs that need a value
have to invent it. Many of these can in fact do without. LILO survives
well without geometry information.

The only case I see where absolutely something is needed is the
case of partitioning an empty disk.
Telling the user at that point: "if you have no idea I'll use
H=255 S=63, that is very often correct, but note that if you also
want to use Windows on this disk it might be better to let Windows
partition first; also, enabling CONFIG_EDD might allow me to guess
what the BIOS uses" may be enough.

Andries

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Szakacsits Szabolcs]

On Fri, 2 Jul 2004, Andries Brouwer wrote:
> On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:
> 
> > Does anybody find the new HDIO_GETGEO semantic useful? Did it help
> > _anything_? 
> 
> Yes. I do.
> 
> There was a steady stream of people reporting on geometry problems.
> All these problems were caused by kernel guessing stuff.

True, I'm well aware that old kernels guessed wrong geometries sometimes,
resulting totally trashed partitions. However far not as much as
currently. Perhaps the steady stream started also when the HDIO_GETGEO
semantic was changed and more people started to use 2.5 and 2.6 kernels.

Two choices to "fix" this guess game:

    1) return error ("I don't know") but providing compatibility
       functionality for things that the kernel knows (e.g. where the
       partition starts). 

    2) use EDD, it does a much better job -- maybe this suggestions
       doesn't make much sense overall, so only 1) left if you don't 
       want to keep guessing.

Instead the HDIO_GETGEO change was a non-sense semantic one, AFAIS. It
doesn't fix anything apparently, only broke _even_ more by not even
guessing just providing some values.

Please correct me if I'm wrong.

> If the kernel no longer volunteers a guess, then we no longer have
> the situation that the guess can be wrong.

The problem is, the kernel still _guesses_ (even potential hard coded
values are guesses). And now it's even more broken than before. 

This is why I asked _what_ it fixed, not from maintaince but from
functionality point of view.

In short, 2.4 was slightly broken, 2.6 is badly broken. Less maintaince
isn't a good argument because deleting the code would have been a much
better solution: it wouldn't break so many things and even if it did,
developers would have notice (function return error that must be always
handled).
 
> The new world is getting much simpler. 

Unfortunately it seems it's more messed up. You didn't write any specific
why the current situation would be better. What does HDIO_GETGEO returns
at present? Hard coded values? Random values? Then why not error instead?

> The only case I see where absolutely something is needed is the
> case of partitioning an empty disk.

Recovery, cloning, ... just unpack a major distro source and grep for
HDIO_GETGEO and you'll see how many things use it for all kind of
purposes.

	Szaka

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[dwm]

On Fri, 02 Jul 2004 18:50:13 +0200, Andries Brouwer wrote:
>On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:
>
>> Please also note, so far nobody stepped forward to fix parted.
>
>Nobody asked, but I wouldnt mind fixing this particular
>aspect of parted.
>
>If nobody else wants to maintain it I can take it, but then
>"maintain" means: zero development, just bugfixes.
>
>Andries

Andries,

  Any chance I could put my hat in the ring, so to speak, to be the 
  maintainer?

++doug

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Here we go again.

Szakacsits Szabolcs <szaka@sienet.hu> writes:

> Two choices to "fix" this guess game:
> 
>     1) return error ("I don't know") but providing compatibility
>        functionality for things that the kernel knows (e.g. where the
>        partition starts). 

As you point out, this will require changes to several userspace
programs.  It is the correct long term solution, but a deprecation
period would be nice.

>     2) use EDD, it does a much better job -- maybe this suggestions
>        doesn't make much sense overall, so only 1) left if you don't 
>        want to keep guessing.

Using EDD to deduce the geometry is the "right" answer.  But this is
sufficiently complex and special-purpose that it has no place in the
kernel.

> Unfortunately it seems it's more messed up. You didn't write any
> specific why the current situation would be better. What does
> HDIO_GETGEO returns at present? Hard coded values? Random values?

Values used by the controller itself.  Also the values you will get
from the "extended INT13" BIOS interface.  As good a geometry as any,
unless you plan to dual-boot Windows.

> Then why not error instead?

Fine idea in the long term, but start by declaring the HDIO_GETGEO
interface "obsolescent" and spit a warning to syslog when it is used.
Finding and fixing all the userspace invocations will take some time.

Right now, HDIO_GETGEO is the only way some applications (e.g., mine)
can convince Parted to use the right geometry.  So, fix Parted to
allow the user (i.e., the higher-level partitioning machinery) to
specify the geometry.  This is the first and last necessary task
before eliminating Parted's use of HDIO_GETGEO.

> On Fri, 2 Jul 2004, Andries Brouwer wrote:
>
> > The only case I see where absolutely something is needed is the
> > case of partitioning an empty disk.
> 
> Recovery, cloning, ...

...moving a drive between machines...

Why does this stupid idea keep coming up?  Inferring the geometry from
the existing partition table is just plain wrong.  It is even more
wrong than the old 2.4 behavior, because it is still a guess, just a
worse guess.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Anton Altaparmakov <aia21@cam.ac.uk> writes:

> On Fri, 2 Jul 2004, Patrick J. LoPresti wrote:
> 
> > Values used by the controller itself.  Also the values you will get
> > from the "extended INT13" BIOS interface.  As good a geometry as any,
> > unless you plan to dual-boot Windows.
> 
> That is not true AFAIK.  EDD reports what the bios reports.

There is more than one BIOS interface.  I call them the "legacy INT13"
interface (INT13/AH=08h) and the "extended INT13" interface
(INT13/AH=48h).  Suggest better terminology if you have any.

The extended interface is also called the "EDD interface".  The Linux
edd.o module queries both this interace and the legacy interface,
leading to some confusion about what "EDD" means in this context.

Try this on a Linux 2.6.7 system.  Do "modprobe edd".  Look in
/sys/firmware/edd/int13_dev80.  In particular compare the contents of
the files named "default_*" with those named "legacy_*".

You will find that the default_* files (i.e., the geometry from the
extended INT13 interface) match the values returned by HDIO_GETGEO.
Which is what I said.

> The HDIO_GETGEO reports crap and certainly not what EDD reports or
> at least not on any(!) PCs I have tried it.

Then you have not tried a SCSI or RAID system.  But this is beside the
point.

> > Right now, HDIO_GETGEO is the only way some applications (e.g., mine)
> > can convince Parted to use the right geometry.  So, fix Parted to
> > allow the user (i.e., the higher-level partitioning machinery) to
> > specify the geometry.  This is the first and last necessary task
> > before eliminating Parted's use of HDIO_GETGEO.
> 
> What?  HDIO_GETGEO does not convince parted to use the right
> geometry!  In 2.6 kernels it convinces parted to use the WRONG
> geometry and screw your data.  That is the whole point!

I keep forgetting that every time I write about this topic, I need to
repeat everything I have said before.

I use a command like this:

  echo bios_cyl:XXX bios_head:YYY bios_sect:ZZZ > /proc/ide/hda/settings

...to fix the values returned by HDIO_GETGEO.  I do this because I
need to partition a blank drive on which I intend to install Windows
(http://unattended.sourceforge.net/).  As I said, this is currently
the only way I can convince Parted to use the correct geometry.

> > Why does this stupid idea keep coming up?  Inferring the geometry from
> > the existing partition table is just plain wrong.  It is even more
> > wrong than the old 2.4 behavior, because it is still a guess, just a
> > worse guess.
> 
> It is not only not wrong but completely correct.  The geometry is
> really something completely made up nowadays.  Nothing really needs
> it at the low level.

Wrong.  The Windows boot loader still reads sectors using the legacy
BIOS interface.  If not for this unfortunate fact, this entire issue
would be moot.

My point, for the 20th time, is that *there is no existing partition
table.*  Why people are suggesting replacing a bad guess with a worse
guess is beyond me.

 - Pat

parted maintainership

[Andries Brouwer]

On Fri, Jul 02, 2004 at 01:28:15PM -0500, dwm@austin.ibm.com wrote:
> On Fri, 02 Jul 2004 18:50:13 +0200, Andries Brouwer wrote:
> >On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:
> >
> >> Please also note, so far nobody stepped forward to fix parted.
> >
> >Nobody asked, but I wouldnt mind fixing this particular
> >aspect of parted.
> >
> >If nobody else wants to maintain it I can take it, but then
> >"maintain" means: zero development, just bugfixes.
> >
> >Andries
> 
> Andries,
> 
>   Any chance I could put my hat in the ring, so to speak, to be the 
>   maintainer?
> 
> ++doug

I do not mind at all.
But parted is not mine. Don't know about the current situation -
always thought of parted as the child of Andrew Clausen.
I have some random source fragments here with a few names,
let me cc them. Probably someone will tell us what is happening.

Andries

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Szakacsits Szabolcs]

On 2 Jul 2004, Patrick J. LoPresti wrote:

> >     2) use EDD, it does a much better job -- maybe this suggestions
> >        doesn't make much sense overall, so only 1) left if you don't 
> >        want to keep guessing.
> 
> Using EDD to deduce the geometry is the "right" answer.  But this is
> sufficiently complex and special-purpose that it has no place in the
> kernel.

Currently no code or a way to do this for all kernels neither in user, 
nor in kernel space, AFAIK. 

Several tools need it. 

Old kernels gave a non-perfect solutions. Today there is a worse approach,
no alternativa and uncontrolled, released, buggy zombie tools are
following what the kernel says: trash the partition table.

Due to the geometry change reported by the kernel and _additional_
partitioning software bugs, sometimes even the layout of existing
partitions are changed, aligned to new, bogus cylinder boundary. Thus not
only Windows but Linux or any other partitions get trashed too in cases.

This kernel geometry change exposed several _different_ partitioning bugs.
 
> Why does this stupid idea keep coming up?  Inferring the geometry from
> the existing partition table is just plain wrong.  It is even more
> wrong than the old 2.4 behavior, because it is still a guess, just a
> worse guess.

In different situations you must use different methods to get the needed
"geometry". 

When you read the geometry from an existing partition table then you
basically don't care about geometry. If it was broken then you won't break
it. If it was good then your values must be good too. But see below.

However there are cases like empty partition table, fixing partition table
if asked, mkntfs, ntfsck, etc when you need the bootloader friendly
geometry what I suspect is the EDD geometry, usually. Linux can't do that:
HDIO_GETGEO doesn't tell and no user space code that could be used for all
kernels.

	Szaka

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Szakacsits Szabolcs]

On 2 Jul 2004, Patrick J. LoPresti wrote:

> You will find that the default_* files (i.e., the geometry from the
> extended INT13 interface) match the values returned by HDIO_GETGEO.

If my memory serves well, you wrote once long ago that your project needs
the "legacy" value to make things work.

Kernel 2.4 guessed usually legacy, right? 

Kernel 2.6 returns extended and it more upsets tools and users with
trashed partitions.

So a simple question: why is returning the extended values better than
returning always the legacy values (or even the previous guess)?

I _do_ know that that won't be perfect either but perhaps it weren't so
broken as it is now. 

BTW, so far nobody answered what the technical benefit returning the
extended values instead of the legacy ones or the previous guess. 
We only know the current values hurt more and there are only better
alternatives, right?

	Szaka

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Bartlomiej Zolnierkiewicz]

On Saturday 03 of July 2004 02:17, Szakacsits Szabolcs wrote:
> On 2 Jul 2004, Patrick J. LoPresti wrote:
> > You will find that the default_* files (i.e., the geometry from the
> > extended INT13 interface) match the values returned by HDIO_GETGEO.
>
> If my memory serves well, you wrote once long ago that your project needs
> the "legacy" value to make things work.
>
> Kernel 2.4 guessed usually legacy, right?

2.4 has drivers/ide/ide-geometry.c which is _exclusively_ for FS use,
IDE driver doesn't need it et all.

> Kernel 2.6 returns extended and it more upsets tools and users with
> trashed partitions.

2.6 returns "raw" IDE CHS not a BIOS translation.

I now think that Andries should have removed HDIO_GETGEO from
IDE driver _completely_ instead of only removing ide-geometry.c.

This can (and should) be still done.

Bartlomiej

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andries Brouwer]

(i) Szaka wrote stuff that I interpreted as stating (a) that parted
is seriously broken, and (b) that parted is unmaintained.

Now that I look, I find parted-1.6.11 with a Changelog with last change,
Andrew Clausen 2004-04-25. That is not long ago.

Upon looking further, I find that Andrew Clausen answered on the
parted mailing list on 2004-06-26, not long ago at all.

So, any suggestions (maybe misunderstood by me) about parted being
unmaintained, are completely incorrect.

Concerning the "seriously broken" part, looking at the various available
sources I find that indeed parted has always been seriously broken,
but now Andrew has added stuff in an attempt to fix things.
His added stuff is broken as well, but clearly work is being done,
so instead of shouting on the kernel list it seems more productive
to tell Andrew precisely in what ways his stuff is broken.
I would expect that very soon parted is fine.


(ii) Various people talk about moving disks between machines.
Such people have not understood the main fact here:
the geometry is not a property of the disk but of the BIOS.
It is futile to hope for a construction that will work across
different machines with different BIOSes.


(iii) Bartlomiej writes:

> I now think that Andries should have removed HDIO_GETGEO from
> IDE driver _completely_ instead of only removing ide-geometry.c.

The main reason that it still exists is that it provides some
information not (easily) available elsewhere, namely the starting
offset.

But it is true, returning 0 in all other fields would have made
it more clear that there is no attempt to return the BIOS geometry.
It might be a good idea to do that.


Andries

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Fri, Jul 02, 2004 at 06:17:53PM +0200, Szakacsits Szabolcs wrote:
>           [kernel related notes are at the end]
> Hello,
> 
> On 2 Jul 2004, Patrick J. LoPresti wrote:
> 
> Parted is looking for (co-)maintainers. Wouldn't you like to be?
> Guillaume? Somebody from SUSE, Red Hat, Debian, anybody? I think 
> it's a real challenge in its current state ;)

I am the current Parted maintainer.  I am rather busy, and haven't had
time to do a good job.  However, I haven't "dropped" it.  I would
welcome a co-maintainer though :)

> > Parted needs a mechanism to let me FORCE the geometry it uses.  Every
> > other partitioning tool has this, usually via command-line switch.

Would this solve any problems?  The people who get hit aren't going
to use the switch, right?

> Currently they blame the BIOS, LILO, GRUB, NTFS, FAT32, Microsoft,
> bootloaders, kernel, hardware, firmware, themself(!!), etc. Parted 
> is so hidden, embedded in tools, installers and behaves so randomly 
> then I think it was very difficult to realise how broken it is, over
> the years.

For the same reasons, I haven't been getting bug reports.  I first found
out that this was a (non-hypothetical) problem on Slashdot.  I can't
reproduce the bug.  Nor can Jeremy Katz at Red Hat.  If anyone out there
can, please get in touch!

> > Having such [geometry] guesswork in Parted itself is a design error,
> 
> Yes. Parted must get the geometry from the partition table unless
>
> 	1) the partition table is empty
> 
> 	2) asked to fix the partition table
> 
> 	3) asked to use user provided values

You need to add these to the list:

	(4) There aren't any partitions that end before cylinder 1024.
	(5) The partition table is inconsistent.

Parted does inspect the partition table to try to figure out the
geometry.  I'm not sure why Parted is messing up so much.  (The current
analyses don't explain it)  I really need more information.

> 1) and 2) need a way to get a "sane" geometry from the BIOS or kernel.

Shouldn't we just use LBA?  (i.e. x/255/63)

> Nevertheless, fixing Parted and all broken tools, that trust the kernel
> getting the most-of-the-time-right-geometry, won't fix the problem
> immediately because nobody can replace all these tools in the wild from
> one day to the other. Transition would take several years.

Is there any evidence of this?  I think 6 months.  (Seriously, has
anyone done any research?)

> Does anybody find the new HDIO_GETGEO semantic useful? Did it help
> _anything_? 

Remember, back when it was proposed, everyone claimed "everyone uses
LBA", and hence it was irrelevant.  I thought it was better to keep it
as it was.

> Because the semantic change did break many people data, installations
> permanently and keeps doing so every day.

I don't understand this sentence.

> Please also note, so far nobody stepped forward to fix parted.

You know there is a patch.  The problem is that "how-to-fix" is
contentious, not that we can't produce a patch.

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Szakacsits Szabolcs]

On Sat, 3 Jul 2004, Andries Brouwer wrote:

> (i) Szaka wrote stuff that I interpreted as stating (a) that parted
> is seriously broken, and (b) that parted is unmaintained.

Right. Parted is definitely looking for (co)maintainers (note, I've never
written parted is unmaintained, even though I think _practically_ it is).

Unfortunately most discussions were in private with FSF/GNU people after I
wrote this to the bug-parted mailing list

  http://lists.gnu.org/archive/html/bug-parted/2004-03/msg00091.html

I'm sure Andrew won't be angry if I quote one of his private emails
(please note he is quite busy with his study, made already heroic efforts
but clearly couldn't/can't focus on the emerging issues).

  Final comment: I think I am inadequate as a maintainer for GNU Parted.
  However, as there are no other volunteers, I am better than nothing.
  Some people have started working on Parted, and I make try to give them
  advice, apply patches, etc.  But, all of these people to date have been
  too busy to take over long-term Parted maintainence.
 
> Concerning the "seriously broken" part, looking at the various available
> sources I find that indeed parted has always been seriously broken,
> but now Andrew has added stuff in an attempt to fix things.
> His added stuff is broken as well, but clearly work is being done,

He was told many many times since last November how to do it correctly.
Even you told him but he still couldn't cope with all the tasks.

> so instead of shouting on the kernel list it seems more productive

Please see subject: "[RFC] Restoring HDIO_GETGEO semantics", it has
nothing to do with parted maintaince.

Fixing parted won't fix all the tools people currently use still trusting
the newer kernels. You can't force everybody to replace immediately their
parted/etc even if they got fixed.

Sorry, I can't explain better. Maybe somebody who gets what I want to say
could translate it to real, understandable English. I'd really appreciate.

> to tell Andrew precisely in what ways his stuff is broken.
> I would expect that very soon parted is fine.

Unfortunately things always went to the wrong direction in the last 8
months. No matter who and how tried to explain.

> (ii) Various people talk about moving disks between machines.

I've seen only Patrick mentioning this.

> Such people have not understood the main fact here:
> the geometry is not a property of the disk but of the BIOS.
> It is futile to hope for a construction that will work across
> different machines with different BIOSes.

Exactly. This is why e.g. ntfsck, mkntfs, recovery and other tools need to
get this value reliable from somewhere. But they can't.

	Szaka

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Fri, Jul 02, 2004 at 02:45:50PM -0400, Patrick J. LoPresti wrote:
> >     2) use EDD, it does a much better job -- maybe this suggestions
> >        doesn't make much sense overall, so only 1) left if you don't 
> >        want to keep guessing.
> 
> Using EDD to deduce the geometry is the "right" answer.  But this is
> sufficiently complex and special-purpose that it has no place in the
> kernel.

You think it should be in user-space?  I don't think talking to the
BIOS should ever be in user-space.

> > > The only case I see where absolutely something is needed is the
> > > case of partitioning an empty disk.
> > 
> > Recovery, cloning, ...
> 
> ...moving a drive between machines...
> 
> Why does this stupid idea keep coming up?  Inferring the geometry from
> the existing partition table is just plain wrong.  It is even more
> wrong than the old 2.4 behavior, because it is still a guess, just a
> worse guess.

Didn't the old 2.4 behaviour include BIOS queries?

In any case, I don't have any evidence that anything is wrong.  On my
computer, I can tell the BIOS to use CHS geometry, (as opposed to
"Auto", "LBA" or "Large") modify the partition table to set the CHS
start/end of the Windows partition to 0, 1024, or anything I like, and
Windows STILL works.  I can't get anything to break!

So, can anyone break Windows?

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Fri, Jul 02, 2004 at 03:57:59PM -0400, Patrick J. LoPresti wrote:
> > It is not only not wrong but completely correct.  The geometry is
> > really something completely made up nowadays.  Nothing really needs
> > it at the low level.
> 
> Wrong.  The Windows boot loader still reads sectors using the legacy
> BIOS interface.  If not for this unfortunate fact, this entire issue
> would be moot.

Have you got any evidence?  Is there any way I can force Windows to
use CHS?

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andries Brouwer]

On Sat, Jul 03, 2004 at 11:35:53AM +1000, Andrew Clausen wrote:

> I would welcome a co-maintainer though :)

Maybe dwm@austin.ibm.com wants to help.

> > > Parted needs a mechanism to let me FORCE the geometry it uses.  Every
> > > other partitioning tool has this, usually via command-line switch.
> 
> Would this solve any problems?

I think that is the wrong question.
This stuff is tricky. Especially when one has several operating systems
on a single disk some degree of control is required. Maybe you say
"in tricky cases use some other partition editor", but it seems
reasonable to offer users the opportunity to specify the desired
geometry.

Of course poor users will listen to the ignorant advice of their
neighbours and do all kinds of things to the geometry in attempts
to solve entirely unrelated problems. The number of support problems
will increase. So, try to supply accurate docs describing the cases
where changing the geometry may be useful, and warning that changing
the geometry may cause the loss of all data on the disk.

Andries


[yes - the discussion forks into many entirely different topics;
other answers in other letters]

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Sat, Jul 03, 2004 at 08:53:39AM +0100, Anton Altaparmakov wrote:
> On Sat, 3 Jul 2004, Andrew Clausen wrote:
> > In any case, I don't have any evidence that anything is wrong.  On my
> > computer, I can tell the BIOS to use CHS geometry, (as opposed to
> > "Auto", "LBA" or "Large") modify the partition table to set the CHS
> > start/end of the Windows partition to 0, 1024, or anything I like, and
> > Windows STILL works.  I can't get anything to break!
> 
> Which version of Windows?

XP home edition (the green box)

> Does it use NTFS as both the boot and system drive?

I am using a single NTFS partition.

Note: I reversed-engineered the Windows FAT bootstrap code.  My analysis
is contained in the file doc/FAT in the Parted source distribution.  I
concluded that Windows uses LBA if the LBA flag is set in the boot
partition table entry.  (i.e. the partition type includes LBA in the
fdisk codes - this corresponds to a bit being set)

> > So, can anyone break Windows?
> 
> Easily.  Modify any of the relevant values in the NTFS bootsector and 
> windows will no longer boot.  So it clearly cares hugely about the 
> geometry.  And at present there is no easy way for us to tell what it is 
> so mkntfs and ntfsclone cannot create bootable partitions on 2.6 kernels.  
> (Works fine on 2.4 using HDIO_GETGEO.)
> 
> The relevant fields are (see linux/fs/ntfs/layout.h or 
> ntfsprogs/include/ntfs/layout.h) in the NTFS_BOOT_SECTOR in the 
> BIOS_PARAMETER_BLOCK:
> 
> u16 sectors_per_track; /* Required to boot Windows. */
> u16 heads;             /* Required to boot Windows. */
> u32 hidden_sectors;    /* Offset to the start of the partition relative 
> to the disk in sectors.  Required to boot Windows. */

I just set the first 2 of these fields to 0, and everything still works.
Am I blessed?  (Or perhaps cursed!)

Isn't hidden_sectors an LBA value (and hence irrelevant to this discussion)?

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Szakacsits Szabolcs <szaka@sienet.hu> writes:

> Currently no code or a way to do this for all kernels neither in user, 
> nor in kernel space, AFAIK. 
>
> Several tools need it. 

Right, which is why putting the logic in Parted is the wrong design.

> However there are cases like empty partition table, fixing partition
> table if asked, mkntfs, ntfsck, etc when you need the bootloader
> friendly geometry what I suspect is the EDD geometry, usually. Linux
> can't do that: HDIO_GETGEO doesn't tell and no user space code that
> could be used for all kernels.

Right.  So this "smart" code has no place in Parted for two reasons:

  1) The best approach varies depending on your kernel.

  2) Parted is not the only tool which needs the Windows-compatible
     geometry.

Parted should do the simplest possible thing.  Allow the user to
specify the geometry, and default it to HDIO_GETGEO (or just
X/255/63).

> If my memory serves well, you wrote once long ago that your project
> needs the "legacy" value to make things work.
>
> Kernel 2.4 guessed usually legacy, right? 

On a blank disk, yes, I believe so.  Or it tried to parse the CMOS
tables or something.  It may have had some "infer from the partition
table" logic, too...  I never looked at it very carefully.

> Kernel 2.6 returns extended and it more upsets tools and users with
> trashed partitions.
>
> So a simple question: why is returning the extended values better
> than returning always the legacy values (or even the previous
> guess)?

Because in 2.6, the extended values are obtained by talking directly
to the IDE controller, not the BIOS.  The "raw controller" geometry
happens to match the extended geometry on every system I have seen.

The problem with having the kernel use the BIOS is complexity; in
particular, the complexity of mapping between BIOS disk numbers and
Linux devices.  This task is "impossible" in theory but quite possible
in practice, and becoming more possible all the time (as more BIOSes
support EDD 3.0).  But it is still complicated, which is why the logic
belongs in user space.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Andries Brouwer <Andries.Brouwer@cwi.nl> writes:

> (ii) Various people talk about moving disks between machines.
> Such people have not understood the main fact here:
> the geometry is not a property of the disk but of the BIOS.

Um...  Yes, right, exactly, very good.  So you *must* obtain the
geometry from the BIOS and not from anything you happen to find on the
disk.  Which is sort of my entire point.

> It is futile to hope for a construction that will work across
> different machines with different BIOSes.

Simply false.  "modprobe edd.o" followed by an examination of the
fields under /sys/firmware/edd will give you precisely the right
values.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Andrew Clausen <clausen@gnu.org> writes:

> > > Parted needs a mechanism to let me FORCE the geometry it uses.
> > > Every other partitioning tool has this, usually via command-line
> > > switch.
> 
> Would this solve any problems?  The people who get hit aren't going
> to use the switch, right?

You are thinking of Parted as an end-user tool.  But there are 1000
times as many people using Parted without even knowing it, every time
they install Linux.

Parted is primarily a component of larger systems; namely, the
RedHat/Suse/etc. installers.  Those larger systems can figure out the
correct geometry (using whatever logic/heuristics/knowledge they have)
and pass it to the tools which need it, of which Parted is just one.

In my case, my software *knows* the geometry because it got it from
/sys/firware/edd.  Right now, I use a hack (/proc/ide/hda/settings) to
override the values returned by HDIO_GETGEO.  I run Parted once to
blow away the partition table, then run it again.  With no partition
table to help it "guess", Parted falls back on the HDIO_GETGEO values,
thus using the geometry I specify.

It works, but I would rather just pass the geometry to Parted when I
run it.

I am suggesting that you cater to the 99.9% case.  This means
providing some way, any way, to override Parted's notion of the
geometry.  In my opinion, you should simply gut the logic for guessing
the geometry, because it really does not belong in Parted.  But I do
not really care as long as I have a way to bypass it.

(Note that this would also provide a way for end users to fix their
partition tables if/when they broke.  Right now, the stock solution
for disks which Parted "broke" is "sfdisk -d | sfdisk -C# -H# -S#".
Wouldn't it be nice if people could use Parted instead?)

> > 1) and 2) need a way to get a "sane" geometry from the BIOS or kernel.
> 
> Shouldn't we just use LBA?  (i.e. x/255/63)

IBM Thinkpads use x/240/63.  In theory, other BIOSes could use
anything.

But x/255/63 is usually a better guess than HDIO_GETGEO.  Except in my
application, which I could fix if Parted had command-line options to
specify the geometry.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Andrew Clausen <clausen@gnu.org> writes:

> On Fri, Jul 02, 2004 at 02:45:50PM -0400, Patrick J. LoPresti wrote:
>
> > Using EDD to deduce the geometry is the "right" answer.  But this
> > is sufficiently complex and special-purpose that it has no place
> > in the kernel.
> 
> You think it should be in user-space?  I don't think talking to the
> BIOS should ever be in user-space.

The kernel already makes the relevant BIOS calls at startup and
stashes the results away (see arch/i386/boot/edd.S).  The edd.o module
exposes these values to userspace under /sys/firmware/edd.

But again, none of this should be Parted's concern.

> > Why does this stupid idea keep coming up?  Inferring the geometry from
> > the existing partition table is just plain wrong.  It is even more
> > wrong than the old 2.4 behavior, because it is still a guess, just a
> > worse guess.
> 
> Didn't the old 2.4 behaviour include BIOS queries?

Or it parsed the CMOS tables directly; I am not sure.  It was a
"guess" in the sense that mapping from BIOS devices to Linux devices
is tricky.

But I still maintain it was a better guess than the current suggestion
of relying on the existing partition table.

> In any case, I don't have any evidence that anything is wrong.  On
> my computer, I can tell the BIOS to use CHS geometry, (as opposed to
> "Auto", "LBA" or "Large") modify the partition table to set the CHS
> start/end of the Windows partition to 0, 1024, or anything I like,
> and Windows STILL works.  I can't get anything to break!
> 
> So, can anyone break Windows?

I believe there are several things going on here, which is why this is
so confusing.

When I first encountered this problem, it was when I used Parted to
create a partition table on a blank disk and then ran the Windows
installer under dosemu.  After the first reboot, I got "NTLDR is
missing".

I believe this was an instance of
<http://support.microsoft.com/?id=314057>, which afflicts FAT
filesystems.  (Installing Windows from DOS starts with a FAT
filesystem which is later converted to NTFS.)

When I forced Parted's notion of the geometry to be the "legacy BIOS"
values and re-created the partition table, the problem went away.
This is what my code does now, and it works fine...  But it relies on
/proc/ide/hdX/settings and HDIO_GETGEO, because Parted has no proper
interface for forcing the geometry.

The Fedora/Suse/Mandrake installer problem is a little different.  It
affects people using NTFS.  But it does not affect everybody.  Here is
my current theory.

We now know that some BIOSes examine the partition table at boot and
*adapt* their notion of the geometry to match.  Yes, really.  You can
partition the drive using some geometry, reboot, query the (legacy)
BIOS interface, and get values which match the partition table.  If
you repeat with a different partition table, you will get a different
legacy geometry from the BIOS.

Not all BIOSes do this.  In fact, most do not.  The report I saw was
from someone using a modern Asus motherboard.

So, when this user installed Fedora Core 2, Parted scrambled the
partition table geometry and the BIOS adapted at the next reboot.  But
the Windows BPB still had the original geometry encoded in it.  Since
it no longer matched the BIOS geometry, the boot loader failed.

In other words, I believe the BPB must match the BIOS geometry, not
necessarily the partition table geometry.  But on some machines, the
BIOS *adapts* to match the partition table, allowing you to break
Windows booting by messing with the partition table.

If you want to get into direct contact with someone actually
experiencing this problem, track down Sean Estabrooks.  He was
collecting reports from Fedora Core 2 users, and he was the one who
noticed that the "legacy" geometry could actually change between
reboots based on the contents of the partition table.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Sat, Jul 03, 2004 at 10:15:47AM -0400, Patrick J. LoPresti wrote:
> Parted is primarily a component of larger systems; namely, the
> RedHat/Suse/etc. installers.  Those larger systems can figure out the
> correct geometry (using whatever logic/heuristics/knowledge they have)
> and pass it to the tools which need it, of which Parted is just one.

Why should they bother?  Shouldn't libparted just do it all for them?
(Shouldn't parted use EDD?)

> I am suggesting that you cater to the 99.9% case.  This means
> providing some way, any way, to override Parted's notion of the
> geometry.  In my opinion, you should simply gut the logic for guessing
> the geometry, because it really does not belong in Parted.  But I do
> not really care as long as I have a way to bypass it.

I was under the impression that 2.6 provides a mechanism for setting the
HDIO_GETGEO thingy... so any program can tell Parted (and everything
else, for that matter) what they want the geometry to be.  Perhaps
I misunderstood your email:

	http://www.uwsg.iu.edu/hypermail/linux/kernel/0404.0/0270.html

It contains this:

	echo "bios_cyl:C bios_head:H bios_sect:S" > /proc/ide/hda/settings

Isn't the kernel the right place for this kind of communication to
be happening, anyway?

> (Note that this would also provide a way for end users to fix their
> partition tables if/when they broke.  Right now, the stock solution
> for disks which Parted "broke" is "sfdisk -d | sfdisk -C# -H# -S#".
> Wouldn't it be nice if people could use Parted instead?)

They can, right?  Just type the above, and then do some dummy thing
in parted.  (Parted doesn't have a "touch" command).

> > > 1) and 2) need a way to get a "sane" geometry from the BIOS or kernel.
> > 
> > Shouldn't we just use LBA?  (i.e. x/255/63)
> 
> IBM Thinkpads use x/240/63.  In theory, other BIOSes could use
> anything.

Do they break on x/255/63?

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Patrick J. LoPresti]

Andrew Clausen <clausen@gnu.org> writes:

> On Sat, Jul 03, 2004 at 10:15:47AM -0400, Patrick J. LoPresti wrote:
> > Parted is primarily a component of larger systems; namely, the
> > RedHat/Suse/etc. installers.  Those larger systems can figure out the
> > correct geometry (using whatever logic/heuristics/knowledge they have)
> > and pass it to the tools which need it, of which Parted is just one.
> 
> Why should they bother?  Shouldn't libparted just do it all for
> them?  (Shouldn't parted use EDD?)

Two reasons:

  1) "...of which Parted is just one."  Whatever logic you fancy for
     determining the geometry, the results need to be available to
     several tools.  Parted is only one such tool; ergo, the logic
     belongs OUTSIDE of it.  Parted should expect to be TOLD the
     geometry in any situation where it matters.

  2) The ideal logic varies depending on the capabilities of your
     kernel.  The distribution vendor knows the capabilities of its
     kernel, and can construct appropriate logic.  Putting logic into
     Parted to handle every possible kernel is a sloppy design.

I hate "smart" software.  Don't be smart; be simple.  Default to
whatever you like, but give me a way to TELL Parted the geometry.

> I was under the impression that 2.6 provides a mechanism for setting
> the HDIO_GETGEO thingy... so any program can tell Parted (and
> everything else, for that matter) what they want the geometry to be.
>
> Perhaps I misunderstood your email:
> 
> 	http://www.uwsg.iu.edu/hypermail/linux/kernel/0404.0/0270.html

You understood correctly, but see below...

> It contains this:
> 
> 	echo "bios_cyl:C bios_head:H bios_sect:S" > /proc/ide/hda/settings
> 
> Isn't the kernel the right place for this kind of communication to
> be happening, anyway?

Not according to the kernel developers.  They are threatening to
remove HDIO_GETGEO completely.

> > (Note that this would also provide a way for end users to fix their
> > partition tables if/when they broke.  Right now, the stock solution
> > for disks which Parted "broke" is "sfdisk -d | sfdisk -C# -H# -S#".
> > Wouldn't it be nice if people could use Parted instead?)
> 
> They can, right?  Just type the above, and then do some dummy thing
> in parted.  (Parted doesn't have a "touch" command).

No, because Parted will "helpfully" infer the geometry from the
existing partition table, no matter what HDIO_GETGEO returns!

In short, the /proc/ide/hdX/settings + HDIO_GETGEO solution 1) only
works on blank drives and 2) uses an interface which the kernel
developers consider a crock.

> > IBM Thinkpads use x/240/63.  In theory, other BIOSes could use
> > anything.
> 
> Do they break on x/255/63?

Yes, absolutely.  This is why I wrote the legacy_* support for the
edd.o module in the first place.  Otherwise, I could have used
x/255/63 and been done with it.

 - Pat

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andrew Clausen]

On Sat, Jul 03, 2004 at 03:40:01PM +0100, Anton Altaparmakov wrote:
> > XP home edition (the green box)
> 
> Hm, I only ever tried XP Pro.

I guess we can try diff'ing the bootstrap code.  Should probably do this
in private for copyright reasons...

> > > Does it use NTFS as both the boot and system drive?
> > 
> > I am using a single NTFS partition.
> 
> I have lots of partitions (mostly Linux, NTFS is at end of disk).

If NTFS is at the end of the disk, doesn't it have to use LBA to address
it?

> > Note: I reversed-engineered the Windows FAT bootstrap code.  My analysis
> > is contained in the file doc/FAT in the Parted source distribution.  I
> > concluded that Windows uses LBA if the LBA flag is set in the boot
> > partition table entry.  (i.e. the partition type includes LBA in the
> > fdisk codes - this corresponds to a bit being set)
> 
> Interesting.  Maybe I don't have this bit set?

This bit only applies to FAT, AFAIK.  There is no corresponding bit
for NTFS.

> > > u16 sectors_per_track; /* Required to boot Windows. */
> > > u16 heads;             /* Required to boot Windows. */
> > > u32 hidden_sectors;    /* Offset to the start of the partition relative 
> > > to the disk in sectors.  Required to boot Windows. */
> > 
> > I just set the first 2 of these fields to 0, and everything still works.
> > Am I blessed?  (Or perhaps cursed!)
> 
> Odd.  Messing up any of the above three values makes my XP Pro fail to 
> boot!

Interesting.  How is your BIOS configured?  (LBA, Auto, CHS or Large?
what CHS values?)

Cheers,
Andrew

Re: [RFC] Restoring HDIO_GETGEO semantics (was: Re: workaround for BIOS / CHS stuff)

[Andries Brouwer]

On Sun, Jul 04, 2004 at 12:45:00AM +1000, Andrew Clausen wrote:

> I was under the impression that 2.6 provides a mechanism for setting the
> HDIO_GETGEO thingy... so any program can tell Parted (and everything
> else, for that matter) what they want the geometry to be.

It is not a good idea to ask the user to tell the kernel so that the kernel
can tell parted. It is easier if one can tell parted directly.

> It contains this:
> 
> 	echo "bios_cyl:C bios_head:H bios_sect:S" > /proc/ide/hda/settings
> 
> Isn't the kernel the right place for this kind of communication to
> be happening, anyway?

No. This is stuff that is going away. It is a kludge that happens to work today.

Andries

Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Szakacsits Szabolcs]

On Sat, 3 Jul 2004, Andries Brouwer wrote:
> 
> But it is true, returning 0 in all other fields would have made
> it more clear that there is no attempt to return the BIOS geometry.
> It might be a good idea to do that.

I fail to see how that would solve _now_ the _current_ serious problem
with HDIO_GETGEO.

There are three different problems.

 1) 2.6 kernels made very visible that the widely used Parted, libparted,
    etc are severely broken. They should be FIXED. Off-topic on linux-kernel.

 2) The semantic change of HDIO_GETGEO severely broke widely used, critical 
    tools. This issue should be HANDLED, preferable as soon as possible. 
    The original thread was supposed to be only about this issue.

 3) There are cases when tools need to invent, not-to-be-discussed-now,
    geometry for different kind of purposes. This should be IMPLEMENTED.

The HDIO_GETGEO facts we have are

    - the new HDIO_GETGEO code seriously broke backward compatibility

    - the old HDIO_GETGEO code still exists, just the values are thrown 
      away, as Andries wrote recently

    - nobody could point out any _technical_ benefit why the new HDIO_GETGEO
      code is better than the old one (the _way_ Andries wanted to push the
      code to user space was quite "unlucky")

    - nobody complained if anything would break if HDIO_GETGEO were restored

    - returning 0 values have an unpredictable impact. Hence perhaps the
      change shouldn't be done in the 2.6 kernels to avoid yet another 
      brown paper bag.

Considering all the above points, it seems logical from practical point 
of view, that the restoration of the old HDIO_GETGEO functionality (or
something that's very close to its behaviour) _temporarily_ for 2.6
kernels makes sense.

Of course this wouldn't mean to be as a fix for the above 1) and 3)
problems. It's the restoration of the user space compatibility _and_
preparation for appropriate HDIO_GETGEO removal.

	Szaka

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Steffen Winterfeldt]

On Mon, 5 Jul 2004, Szakacsits Szabolcs wrote:

> On Sat, 3 Jul 2004, Andries Brouwer wrote:
> > 
> > But it is true, returning 0 in all other fields would have made
> > it more clear that there is no attempt to return the BIOS geometry.
> > It might be a good idea to do that.
> 
> I fail to see how that would solve _now_ the _current_ serious problem
> with HDIO_GETGEO.

It wouldn't, the damage has been done. I just wish HDIO_GETGEO had been
removed rather than changed.

>  1) 2.6 kernels made very visible that the widely used Parted, libparted,
>     etc are severely broken. They should be FIXED. Off-topic on linux-kernel.

It's not *severly* broken. All partitioning tools use some kind of
heuristics and it's just a minor variation compared to, say, fdisk that
makes parted fail.

> Considering all the above points, it seems logical from practical point 
> of view, that the restoration of the old HDIO_GETGEO functionality (or
> something that's very close to its behaviour) _temporarily_ for 2.6
> kernels makes sense.

It's too late. Rather than updating the kernel you could as well update your
favorite partitioning tool.

Changing the semantics of HDIO_GETGEO either way in the (supposedly) stable
2.6.x series it IMHO not a good idea.


Steffen

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Andries Brouwer]

On Mon, Jul 05, 2004 at 02:14:50PM +0200, Szakacsits Szabolcs wrote:

> There are three different problems.
> 
>  1) 2.6 kernels made very visible that the widely used Parted, libparted,
>     etc are severely broken. They should be FIXED. Off-topic on linux-kernel.

Indeed. Andrew's recent patch greatly improved the situation.
Recently we exchanged half a dozen letters. I have good hopes
that in the near future *parted will improve a bit more.

>  2) The semantic change of HDIO_GETGEO severely broke widely used, critical 
>     tools. This issue should be HANDLED, preferable as soon as possible. 
>     The original thread was supposed to be only about this issue.

Well. In case you reveal precisely which tools you are thinking of,
I am quite willing to contribute what I can to improve them.

>  3) There are cases when tools need to invent, not-to-be-discussed-now,
>     geometry for different kind of purposes. This should be IMPLEMENTED.

Again. You shout with capital letters, but it is more productive to do.
In case you reveal the purposes not-to-be-discussed-now of these anonymous
tools, then again, I am quite willing to contribute what I can.
My email address is aeb@cwi.nl. Maybe parted is being worked on.
We can discuss everything else that you think needs fixing.

> Considering all the above points, it seems logical from practical point 
> of view, that the restoration of the old HDIO_GETGEO functionality (or
> something that's very close to its behaviour) _temporarily_ for 2.6
> kernels makes sense.
> 
> Of course this wouldn't mean to be as a fix for the above 1) and 3)
> problems. It's the restoration of the user space compatibility _and_
> preparation for appropriate HDIO_GETGEO removal.

You have not convinced me. The change is over two years old.
Let us first discuss all these tools that may need fixing.

Andries


[Note that as it turns out there are situations involving a RAID
where one needs to know where the last cylinder starts. Funny.
This forces the kernel to have ideas about geometry at least for
that case.]

[Note that we saw last week that nowadays one meets FAT filesystems
with zero geometry fields that are reported to work fine with
Windows XP. Also Microsoft is trying to get away from this
geometry nonsense.]

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Bartlomiej Zolnierkiewicz]

On Monday 05 of July 2004 14:14, Szakacsits Szabolcs wrote:

>     - the old HDIO_GETGEO code still exists, just the values are thrown
>       away, as Andries wrote recently

No, the code (ide-geometry.c) is gone.

>     - nobody could point out any _technical_ benefit why the new
> HDIO_GETGEO code is better than the old one (the _way_ Andries wanted to

Andries pointed it many times but you seem to completely ignore it
(see comments in ide-geometry.c):

 * I did this, but it doesn't work - there is no reasonable way to find the
 * correspondence between the BIOS numbering of the disks and the Linux
 * numbering. -aeb
 *
 * The code below is bad. One of the problems is that drives 1 and 2
 * may be SCSI disks (even when IDE disks are present), so that
 * the geometry we read here from BIOS is attributed to the wrong disks.
 * Consequently, also the former "drive->present = 1" below was a mistake.
 *
 * Eventually the entire routine below should be removed.

I also pointed out that IDE driver _doesn't_ need BIOS geometry et all.

> push the code to user space was quite "unlucky")

Yep.

>     - nobody complained if anything would break if HDIO_GETGEO were
> restored
>
>     - returning 0 values have an unpredictable impact. Hence perhaps the
>       change shouldn't be done in the 2.6 kernels to avoid yet another
>       brown paper bag.

Yes, it won't help.

We need new ioctl for a get_start_sect() (first sector of a partition).

> Considering all the above points, it seems logical from practical point
> of view, that the restoration of the old HDIO_GETGEO functionality (or
> something that's very close to its behaviour) _temporarily_ for 2.6
> kernels makes sense.

We can restore ide-geometry.c or try to return values obtained from
EDD code through IDE driver.  Alternatively we can add new ioctl for
start of partition and remove HDIO_GETGEO from IDE driver completely
but probably it is too late for this for 2.6 (we should do it early
in 2.7 then).  Andries?

Bartlomiej

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Andries Brouwer]

On Mon, Jul 05, 2004 at 03:13:48PM +0200, Bartlomiej Zolnierkiewicz wrote:

> We can restore ide-geometry.c or try to return values obtained from
> EDD code through IDE driver.  Alternatively we can add new ioctl for
> start of partition and remove HDIO_GETGEO from IDE driver completely
> but probably it is too late for this for 2.6 (we should do it early
> in 2.7 then).  Andries?

For Linux purposes geometry is almost irrelevant.
(But, as I remarked in another letter, some RAIDs need something.)

This means as a first approximation that "geometry" is not a kernel
business. Certainly the present discussion is not about the ide-driver.
Indeed, what people want is the geometry that a certain BIOS will assign
to a disk. That depends on the BIOS, and on whether the user has
set things up with Normal / Large / LBA in the BIOS setup.
(In case the disk was left out of the BIOS setup, this particular
concept of geometry does not exist at all.)

So, the question is whether information from the BIOS can be
exposed. Well, that is not impossible, and EDD already does this.

Why would people want to know what a BIOS would do?
So far I have mostly seen two classes of wishes.
One is to install lots of new Windows systems on blank disks
- that is done faster and more conveniently from Linux -
where there is no partition table yet to inspect.
The other is to create or modify NTFS filesystems.

Such are reasonable wishes, but rather special purpose.
For the time being I have good hopes that it will turn out
to be possible to do these things. Maybe using present EDD.
Maybe by extending EDD a little.

Five years ago I wrote a library that takes collected BIOS info,
and collected Linux info, and tries to match BIOS disks with
Linux disks. Of course it is impossible to do this right, but
one can find heuristics that often work. Such heuristics do not
belong in the kernel, but a user space application can decide,
given the known situation, whether a guess is probably right,
or perhaps consult the user.

I am awaiting the discussion with Szaka.

Andries

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Szakacsits Szabolcs]

On Mon, 5 Jul 2004, Bartlomiej Zolnierkiewicz wrote:
> On Monday 05 of July 2004 14:14, Szakacsits Szabolcs wrote:
> 
> >     - nobody could point out any _technical_ benefit why the new
> > HDIO_GETGEO code is better than the old one (the _way_ Andries wanted to
> 
> Andries pointed it many times but you seem to completely ignore it

Hmmm. I'm recovering people's partition tables in my spare time,
voluntarily, free of charge when they got trashed due to Andries' and
Andrew's bugs over the last two years (gpart, testdisk and parted's 
rescue mode don't always work),

     http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html#troubleshoot

I reported them several bugs, hints, reasons, potential reasons, guesses,
user feature request both privately and publicly.

I do know very well Andries' arguments, I've learnt them the hard way.
Actually I responded them several times, even in this thread.

> I also pointed out that IDE driver _doesn't_ need BIOS geometry et all.

Thanks but I thought it was off-topic and think the same now, too. It was
explained several times.

However none of you who responded seems to understand, still, what I want
to say.

You can't fix, for example, parted 1.6.11 and all earlier versions when
one does a 2.4 -> 2.6 kernel upgrade. If a user uses an old enough tool on
the new kernels then it can trash its partition table whatever the OS it
is (not only the geometry but the layout in sectors, too).

Also, sometimes [even very popular] distros ship one or two old tools, no
need for kernel upgrade. Whenever a user use the shipped old tool on 2.6
it can trash the partition table, being during install or later.

You, Bartlomiej Zolnierkiewicz, Andries Brouwer and Steffen Winterfeldt
say don't care about those people. OK, at least now it's documented
and this thread can be pointed out as a reference in the future.

Since there is nothing I could do more if maintainers aren't willing to
fix their more destructive 2.6 kernel code, the case is closed from my
part by this email.

Thanks,
	Szaka

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Bartlomiej Zolnierkiewicz]

On Monday 05 of July 2004 20:09, Szakacsits Szabolcs wrote:
> On Mon, 5 Jul 2004, Bartlomiej Zolnierkiewicz wrote:
> > On Monday 05 of July 2004 14:14, Szakacsits Szabolcs wrote:
> > >     - nobody could point out any _technical_ benefit why the new
> > > HDIO_GETGEO code is better than the old one (the _way_ Andries wanted
> > > to
> >
> > Andries pointed it many times but you seem to completely ignore it
>
> Hmmm. I'm recovering people's partition tables in my spare time,
> voluntarily, free of charge when they got trashed due to Andries' and
> Andrew's bugs over the last two years (gpart, testdisk and parted's
> rescue mode don't always work),
>
>      http://mlf.linux.rulez.org/mlf/ezaz/ntfsresize.html#troubleshoot
>
> I reported them several bugs, hints, reasons, potential reasons, guesses,
> user feature request both privately and publicly.
>
> I do know very well Andries' arguments, I've learnt them the hard way.
> Actually I responded them several times, even in this thread.

OK

> > I also pointed out that IDE driver _doesn't_ need BIOS geometry et all.
>
> Thanks but I thought it was off-topic and think the same now, too. It was
> explained several times.

How can this be off-topic?  In case of IDE disks HDIO_GETGEO is handled by
ide-disk driver.  In 2.4 there was ide-geometry.c file (part of IDE driver)
which contained code for retrieving BIOS geometry (Andries removed it in 2.5),
Restoring 2.4 way of handling HDIO_GETGEO requires changes to the IDE driver.

> However none of you who responded seems to understand, still, what I want
> to say.
>
> You can't fix, for example, parted 1.6.11 and all earlier versions when
> one does a 2.4 -> 2.6 kernel upgrade. If a user uses an old enough tool on
> the new kernels then it can trash its partition table whatever the OS it
> is (not only the geometry but the layout in sectors, too).
>
> Also, sometimes [even very popular] distros ship one or two old tools, no
> need for kernel upgrade. Whenever a user use the shipped old tool on 2.6
> it can trash the partition table, being during install or later.

I understand this perfectly and I'm thinking about the best way to solve it.

> You, Bartlomiej Zolnierkiewicz, Andries Brouwer and Steffen Winterfeldt
> say don't care about those people. OK, at least now it's documented
> and this thread can be pointed out as a reference in the future.

Calm down because your false accusations are also documented now. ;-)

> Since there is nothing I could do more if maintainers aren't willing to
> fix their more destructive 2.6 kernel code, the case is closed from my
> part by this email.

Bartlomiej

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Bartlomiej Zolnierkiewicz]

Andries, the question was "What should we do with HDIO_GETGEO breakage?"
not "Why does somebody need the BIOS geometry?". :-)

We can fix HDIO_GETGEO to behave like in 2.4 or remove it (preferable),
current situation is bad.

On Monday 05 of July 2004 16:00, Andries Brouwer wrote:
> On Mon, Jul 05, 2004 at 03:13:48PM +0200, Bartlomiej Zolnierkiewicz wrote:
> > We can restore ide-geometry.c or try to return values obtained from
> > EDD code through IDE driver.  Alternatively we can add new ioctl for
> > start of partition and remove HDIO_GETGEO from IDE driver completely
> > but probably it is too late for this for 2.6 (we should do it early
> > in 2.7 then).  Andries?
>
> For Linux purposes geometry is almost irrelevant.
> (But, as I remarked in another letter, some RAIDs need something.)
>
> This means as a first approximation that "geometry" is not a kernel
> business. Certainly the present discussion is not about the ide-driver.
> Indeed, what people want is the geometry that a certain BIOS will assign
> to a disk. That depends on the BIOS, and on whether the user has
> set things up with Normal / Large / LBA in the BIOS setup.
> (In case the disk was left out of the BIOS setup, this particular
> concept of geometry does not exist at all.)
>
> So, the question is whether information from the BIOS can be
> exposed. Well, that is not impossible, and EDD already does this.
>
> Why would people want to know what a BIOS would do?
> So far I have mostly seen two classes of wishes.
> One is to install lots of new Windows systems on blank disks
> - that is done faster and more conveniently from Linux -
> where there is no partition table yet to inspect.
> The other is to create or modify NTFS filesystems.
>
> Such are reasonable wishes, but rather special purpose.
> For the time being I have good hopes that it will turn out
> to be possible to do these things. Maybe using present EDD.
> Maybe by extending EDD a little.
>
> Five years ago I wrote a library that takes collected BIOS info,
> and collected Linux info, and tries to match BIOS disks with
> Linux disks. Of course it is impossible to do this right, but
> one can find heuristics that often work. Such heuristics do not
> belong in the kernel, but a user space application can decide,
> given the known situation, whether a guess is probably right,
> or perhaps consult the user.
>
> I am awaiting the discussion with Szaka.
>
> Andries

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Andries Brouwer]

On Mon, Jul 05, 2004 at 09:05:05PM +0200, Bartlomiej Zolnierkiewicz wrote:

> Andries, the question was "What should we do with HDIO_GETGEO breakage?"
> not "Why does somebody need the BIOS geometry?". :-)
> 
> We can fix HDIO_GETGEO to behave like in 2.4 or remove it (preferable),
> current situation is bad.

I don't know precisely why.

Neither of your two proposed actions appeals to me.

Here is an ioctl, and it is used for legitimate purposes
(finding the starting offset of a partition).
You cannot remove it. We can think again in 2.7.
For now, leave the kernel interface constant.

Is there any advantage in going back? I don't think so.
The old situation was broken. People lost all data.

Also "the old situation" is badly defined. The returned value differs
for 2.0, 2.2, 2.4, 2.6.

No. We must go forward.

Now distributions can take care of themselves. They can patch the kernel
as they like, or patch parted as they like, or do any number of other things.
RedHat and SuSE can take their own decisions. With some luck there is a new
parted next week or so that they could offer.

So we can go slowly and quietly, investigate precisely what happens, and why
and how such things can be fixed. I think I understand rather well what is
(was) wrong with parted. Maybe Szaka can teach me about other tools that are
broken. I am confident that we can fix them, maybe in hours rather than days.

As a side result we will have something valuable, namely standard software
that tries to handle BIOS data. Several orders of magnitude more reliable
than our old guesses.

Andries

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Bartlomiej Zolnierkiewicz]

On Monday 05 of July 2004 23:08, Andries Brouwer wrote:
> On Mon, Jul 05, 2004 at 09:05:05PM +0200, Bartlomiej Zolnierkiewicz wrote:
> > Andries, the question was "What should we do with HDIO_GETGEO breakage?"
> > not "Why does somebody need the BIOS geometry?". :-)
> >
> > We can fix HDIO_GETGEO to behave like in 2.4 or remove it (preferable),
> > current situation is bad.
>
> I don't know precisely why.
>
> Neither of your two proposed actions appeals to me.
>
> Here is an ioctl, and it is used for legitimate purposes
> (finding the starting offset of a partition).

It is also _abused_ for less legitimate purposes.

> You cannot remove it. We can think again in 2.7.
> For now, leave the kernel interface constant.

We can at least consider adding BLKPARTSTART and deprecating
HDIO_GETGEO so people at least will know that they should upgrade.

> Is there any advantage in going back? I don't think so.
> The old situation was broken. People lost all data.

According to szaka the "the new situation" is similar. :(

> Also "the old situation" is badly defined. The returned value differs
> for 2.0, 2.2, 2.4, 2.6.

This is just sick: same ioctl - different returned values.
Please never ever do it again - we can avoid such problems in future.

> No. We must go forward.

Yep, 2.7.1 should remove HDIO_GETGEO.
We should have done this in 2.5, really.

> Now distributions can take care of themselves. They can patch the kernel
> as they like, or patch parted as they like, or do any number of other
> things. RedHat and SuSE can take their own decisions. With some luck there
> is a new parted next week or so that they could offer.

What about people running old parted and new vanilla kernels?

> So we can go slowly and quietly, investigate precisely what happens, and
> why and how such things can be fixed. I think I understand rather well what
> is (was) wrong with parted. Maybe Szaka can teach me about other tools that
> are broken. I am confident that we can fix them, maybe in hours rather than
> days.

What about people using old versions of these tools (if any).

What I'm only worried is that there might be cases when 2.4 worked
and 2.6 doesn't which with combination with bugs in the tools cause
"where is my data?" issue.

> As a side result we will have something valuable, namely standard software
> that tries to handle BIOS data. Several orders of magnitude more reliable
> than our old guesses.

Yep.

Bartlomiej

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Szakacsits Szabolcs]

On Mon, 5 Jul 2004, Bartlomiej Zolnierkiewicz wrote:
> On Monday 05 of July 2004 23:08, Andries Brouwer wrote:
> 
> > Is there any advantage in going back? I don't think so.
> > The old situation was broken. People lost all data.

Yes they did and still do for example because of the reasons Bartlomiej
quoted and 

	http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-10/4682.html

but please explain what you mean exactly "they lost data"? There are many
ways to lose data and they have different severities.

I've meant partition table corruptions. When the values returned by
HDIO_GETGEO don't match what are in the partition table, [lib]parted will
rewrite all partition entries. "Fixes" them as Andrew says. This "fix"
breaks bootability if some bootloaders rely on them and in some cases
the partition starts are even moved to wrong cylinder boundaries due to
additional problems in tools using [lib]parted.
 
> According to szaka the "the new situation" is similar. :(

No. The situation in 2.6 is much more serious. Using 2.4 kernels people
lost data due to partition corruptions, but not so often, perhaps 1-2
reports a week. Using 2.6, several a day.

Are you really not aware of the seriousness of the issue? Didn't you read
the bugzilla URL's sent? LWN, Eweeks, O'Reilly, OSNews, Slashdot, Amazon
and many others discussed this 2.6 kernel problem already. Only kernel
developers aren't aware of it? :-o

> > Also "the old situation" is badly defined. The returned value differs
> > for 2.0, 2.2, 2.4, 2.6.

2.4 was slightly broken. 2.6 is much more broken than 2.4 from the
number of partition table corruptions point of view.
 
> What I'm only worried is that there might be cases when 2.4 worked
> and 2.6 doesn't which with combination with bugs in the tools cause
> "where is my data?" issue.

No need to worry ;) This is a known fact for about 6-7 weeks, at least.
But it was also reported here almost every months since last October. Now
that some major distros with 2.6 started to ship, the 2.6 HDIO_GETGEO
breakage got really confirmed.

That's why I asked several times, what did 2.6 HDIO_GETGEO fix? 

When does 2.6 work when 2.4 doesn't? 

I'm not aware of any. Nobody wrote any. 

Andries just avoids straight answers for touchy and detailed questions.

	Szaka

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Andries Brouwer]

On Tue, Jul 06, 2004 at 02:17:47AM +0200, Szakacsits Szabolcs wrote:
...

We agree about most of the facts, and I am too lazy to quarrel anyway.
Such a waste of time. But we draw different conclusions.

We discover that the present combination of 2.6 and parted is
unfortunate. Because of bugs in parted. OK - so lets fix them.
There is nothing especially difficult there.

You prefer a larger change to the kernel above a smaller change
to parted, where the change to parted fixes parted, and the change
to the kernel makes the kernel buggier. I do not.

You say that other utilities are affected. Maybe, yes.
So let us look at them. In the time spent writing all these
letters we probably could have fixed a few.

Andries

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Steffen Winterfeldt]

On Tue, 6 Jul 2004, Szakacsits Szabolcs wrote:

> On Mon, 5 Jul 2004, Bartlomiej Zolnierkiewicz wrote:
> > On Monday 05 of July 2004 23:08, Andries Brouwer wrote:

[...]

> Are you really not aware of the seriousness of the issue? Didn't you read
> the bugzilla URL's sent? LWN, Eweeks, O'Reilly, OSNews, Slashdot, Amazon
> and many others discussed this 2.6 kernel problem already. Only kernel
> developers aren't aware of it? :-o
> 
> > > Also "the old situation" is badly defined. The returned value differs
> > > for 2.0, 2.2, 2.4, 2.6.
> 
> 2.4 was slightly broken. 2.6 is much more broken than 2.4 from the
> number of partition table corruptions point of view.

Neither 2.6 nor 2.4 are broken. It was well known all the time that
HDIO_GETGEO returns CHS values that are close to useless: all partitioning
tools spend lots of time on partition table guesswork. I really don't see a
point in blaming 2.6 here.

Nevertheless we have the fact that parted and kernel 2.6 don't work well
together. Now what? To get out of this people have to get either a changed
parted or a changed kernel.

And IMO it's far better to fix parted than to reintroduce an obscure piece
of code into the kernel.


Steffen

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Szakacsits Szabolcs]

On Tue, 6 Jul 2004, Andries Brouwer wrote:

> We agree about most of the facts, and I am too lazy to quarrel anyway.

I only waited for straight, technical answers what you always avoided.
When you answer, you repeat off-topic or too general things. This is not
enough to make a competent, responsible decision to soften the damage 2.6
kernels cause.

> Such a waste of time. But we draw different conclusions.

Please point out where I draw a conclusion? When I wrote RFC (request for
comments)? Or "restoring" (implying attention for discussion). I've never
written "restore!" (imperative mood).

Or when you or anybody else failed to give any reasonable technical answer
why not to restore, I wrote (unfortunately in pretty broken English, but I
hope it was still understandable, so no change now either),

  "it seems logical from practical point of view, that the restoration of
   the old HDIO_GETGEO functionality (or something that's very close to
   its behaviour) _temporarily_ for 2.6 kernels makes sense."

In this context, the important words are "seems" and "make sense". It's
still a suggestion based on the collected information. At least that's
what I've meant.

Explanation: it's pretty impossible to make any conclusion when you don't
provide straight answers. 

> We discover that the present combination of 2.6 and parted is
> unfortunate. Because of bugs in parted. 

And because of the 2.4 -> 2.6 HDIO_GETGEO _semantic_ change.

> OK - so lets fix them.

That's what people are waiting for over 8 months, at least. Both needs
handling, not only one.

> You prefer a larger change to the kernel above a smaller change
> to parted, 

These are your words, again, not mine. Read here what I wrote and if you
still don't understand, no problem I'll try to write it down a different
way,

   http://groups.google.com/groups?selm=2eAC2-7Zf-7%40gated-at.bofh.it

At the same time I also apologize for the tree separate, big capital
words. I've meant them to emphasize visually, too, that there are
fundamentally _three_ different things to solve. If one shouts, then 
HE USUALLY CAPITALIZES MANY WORDS NEXT TO EACH OTHER. But I've never 
did, never will! Not even in this email for demonstration! ;)

So, please check the email out again because you answered the two
off-topic problems and gave an off-topic answer for the on-topic one. 
I also recommend going through the old emails by reading them much more
carefully and then thinking about them a bit. Please also don't forget 
to check out the mentioned URL's, links, reading user comments.

Of course, only if you care about the issue on the _kernel_ side.

> the change to the kernel makes the kernel buggier.

This was your most productive comment so far in the subject ;)

Why does it make the kernel buggier? Please detail. Deeply, technically.
Many of us would greatly appreciate. Seriously. But not the general, too
high level way, you explained over the last two years and to Jeff last
time. That's not enough in the current situation.

You mentioned a RAID that needs the 2.6 geometry. What's that? Would it
break by the change? Could it be fixed? Could something else break?

Why don't you even consider the temporary restoration when you keep saying
geometry doesn't exist, geometry doesn't matter? If it didn't matter then
it can be anything, so why not the one that doesn't break compatibility?
We would have much less data loss, just as with 2.4 kernels.

What would happen if it returns 0? How current [lib]parted handles
it? Other things? 

> You say that other utilities are affected. Maybe, yes. So let us look
> at them. In the time spent writing all these letters we probably could
> have fixed a few.

Sorry, off-topic. This thread is not about patching utilities. Please keep
the subject.

	Szaka

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Andries Brouwer]

On Tue, Jul 06, 2004 at 08:56:22PM +0200, Szakacsits Szabolcs wrote:

> > You say that other utilities are affected. Maybe, yes. So let us look
> > at them. In the time spent writing all these letters we probably could
> > have fixed a few.
> 
> Sorry, off-topic. This thread is not about patching utilities. Please keep
> the subject.

Pity. There are still a few days. Soon I'll be in Korea and Russia and
elsewhere, and before you know it it'll be September again. This week
there are a few days to do some Linux-related work.

Since you stressed so loudly that stuff was broken, I would have liked
to learn precisely what, and to fix, if possible.  Maybe it was not important.


> What would happen if it returns 0?

There is no serious suggestion to do so. It is 2.6 and interfaces are stable.
Also, this much maligned output still contains some information.
Making it 0 would have been a good signal around 2.5.6.


> You mentioned a RAID that needs the 2.6 geometry.

No, you misunderstood. It is just that the blanket statement
"geometry plays no role in Linux" is only true "roughly speaking",
but there are details. One such detail is that certain Promise firmware
expects a RAID superblock on the first sector of the last track of
the last full cylinder, IIRC. Where is that? That is known only
when a geometry has been defined.
(There was a discussion, forgot the details, see e.g.
http://www.uwsg.iu.edu/hypermail/linux/kernel/0307.2/0917.html )
See also the code in 2.4 pdcraid.c.

So, both you and I were telling Bartlomiej: "No, we are not talking about
the ide driver", as indeed we were not, but there may still be some
relevance even for drivers/ide.


> Why does it make the kernel buggier?
> Please detail.


There is a tension between correctness and convenience.

If an installer sees a partition with swap signature, then that is swap space,
isn't it? No need to ask the user. Only, it was my FreeBSD partition that
earlier had been swap space. Bye bye.

Everywhere where magic numbers are involved, where there is "recognition",
there are bugs. Low probability bugs. If a user knows the system, when
it is clear what is happening, then recognition is convenient, and in the
rare cases that something is misrecognized the problem is easily solved.

But when systems get more complicated, many layers of software, and on all
levels there is "recognition", then completely mysterious bugs will happen
with very low probability.

The conclusion is that a good system must be absolutely correct on the low levels.
No guessing. Of course heuristics also occur on low levels, but there
the function must be to improve efficiency. The punishment for guessing wrong
may be to spend a few milliseconds more, but not to get the wrong answer.

The Linux kernel started as a plaything but is getting complicated.
Long ago is was completely reasonable to guess what type partition table
and what type rootfilesystem the user had. There were only a few possibilities,
mostly easily recognized. But the number of possibilities grows, and the number
of users grows so that even unlikely things really occur. We got rootfstype=.
Also the "reread partition table" must not guess but get the type specified.

So, that was the general talk that you do not want to hear.

One of the many places where we guess is for the return values of HDIO_GETGEO.
What did it return? The precise description would fill several pages, but
the CMOS was inspected (only on i386, only for the first two disks),
the partition table was inspected (depending on the phase of the moon,
the IDE or SCSI driver used), switch settings on cards, as far as these
were readable, the IDENTIFY report from the disk, arbitrary numbers
like 15, 16, 63, 255 were invented, special cases for disk managers ...

How does a monster like that arise? Wart upon wart. Well, we guess, and
usually right, but not always, then invent a correction to guess a bit
better in some special situations, then ...
And times change, and likely guesses become less likely, and changes are made ...

At some point in time the monster must be eliminated.

Fortunately, the kernel does not need the result of this guesswork any longer.
Suppose the kernel exports all information it has to user space
without extracting guesses, and leaves the guessing to user space.
Then the kernel is 100% correct and well-defined.
That is what we like.

And even better, user space can do more than the kernel did.
You see that already in the current parted. It looks not only at
the partition table, but also inside FAT and NTFS filesystems
to inspect them for geometry info.
Looks like it may become better than any combination of old parted
with some kernel ever was.

So that is my answer. Ugly code, obscure and buggy guesswork is
being eliminated from the kernel. Rejoice!

Now this happened first in 2.5.6 if I recall correctly, over two
years ago, and a few programs had to be adapted a little.
I think fdisk, cfdisk, sfdisk, probably also lilo, were adapted rather
quickly, but, as we discovered recently, parted took more time. Ach.

Only a rather small amount of geometry code was removed, roughly speaking
a single file, and a much larger cleanup will follow, I hope.
This means that it would not be especially difficult to go back.
There is no reason to do so.


Andries

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Roman Zippel]

Hi,

On Wed, 7 Jul 2004, Andries Brouwer wrote:

> How does a monster like that arise? Wart upon wart. Well, we guess, and
> usually right, but not always, then invent a correction to guess a bit
> better in some special situations, then ...
> And times change, and likely guesses become less likely, and changes are made ...
> 
> At some point in time the monster must be eliminated.
>
> [..]
> 
> Now this happened first in 2.5.6 if I recall correctly, over two
> years ago, and a few programs had to be adapted a little.
> I think fdisk, cfdisk, sfdisk, probably also lilo, were adapted rather
> quickly, but, as we discovered recently, parted took more time. Ach.

I basically agree with your argumentation, but it wasn't really eliminated 
in 2.5.6. The kernel still provides some values to the users. If the 
kernel doesn't know, it should clearly say so, this is where we fucked up. 
Silently fixing a few applications and leaving everybody else believing 
everything is well doesn't help.
At this point we either complete the job and remove this ioctl or we 
restore the 2.4 behaviour (maybe with a deprecated warning).

bye, Roman

Re: Restoring HDIO_GETGEO semantics for 2.6 (was: Re: [RFC] Restoring HDIO_GETGEO semantics)

[Szakacsits Szabolcs]

On Wed, 7 Jul 2004, Roman Zippel wrote:

> At this point we either complete the job and remove this ioctl or we 
> restore the 2.4 behaviour (maybe with a deprecated warning).

Well, yes. Perhaps a competent guy/gal could even fix most of the broken
2.4 cases during the same time, e.g. by using EDD, if possible and make
sense. But somehow I doubt anybody would take this nasty retore&fix
challange and actually it's even possible.

I also say, things might rely on the 2.6 behavior now, thus they might be
broken by the restoration. They should be investigated to prevent
deserving another brown paper bag.

Andries says, it's not a kernel issue because he is going on vacation soon
(that's why he's off-topic all the time, wanting to adjust only the easy
user space quickly).

	Szaka