[RFC] Platform information services

[RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2859 bytes --]

Hi there everybody,

I'm opening the RFC because I want to add some kind of infrastructure to
retrieve the address of system/platform structures. I will explain
myself: my use case is in i386-pc and for the drivemap module, in which
a function installs a TSR int13h handler. This requires the function to
have access to two real mode structures, namely the BIOS Data Area,
which is based at 0040:0000h; and the Interrupt Vector Table, which
conventionally starts at 0 but that could have been placed elsewhere by
the use of the LIDT instruction.

Currently, the code just "hopes for the best" and accesses 0x0 and 0x400
directly as protected mode addresses from within the GRUB environment,
which has the additional assumption that GRUB has not enabled any kind
of paging or memory mapping. Obviously, the Right Way for this would be
for the code to check where its targets are, but even when the location
of the IVT can be queried by the non-privileged instruction SIDT, that
would require a privileged trip to real mode and back from it in order
to query the location of the real mode IVT instead of the pmode IDT that
is in effect in the GRUB environment (but this still does not deal with
a possible paging).

Thus, the best course of action I see would be to add a "platform
information" infrastructure in kernel (that last word has probably put
me on the death list of several people here ¬¬). My idea would add a
cross-platform header platform.h declaring the following prototype:

void* grub_machine_get_platform_structure (int stidx);

Then, the machine.h file for each platform would declare which
structures are available, like the IVT and the BDA for i386-pc. The code
would be used like this:

#include <grub/platform.h>
#include <grub/machine/machine.h>

/* (...) Now from within a function */
grub_uint8_t* bda = (grub_uint8_t*)grub_machine_get_platform_structure
(GRUB_MACHINE_I386_BDA);

if (!bda)
  /* Error info in errno & errmsg - bail out */
else
  grub_printf ("Avail mem: %d KiB", *((grub_uint16_t*)(bda + 0x13)));


The pointers provided by this new function would be guaranteed to:
  - Be able to access the whole requested structure (if any segmentation
is in effect, the whole struct is addressable from that base address)
  - Have been mapping-marshaled, i.e. if any kind of paging has been
enabled, the address returned can be directly used as a C pointer.
  - Be read-write? Maybe this could be requested through an additional
"flags" parameter to the function...

Initially this addition would only benefit i386-pc and in particular my
drivemap patch, but maybe it can also be used by modules to query EFI
info structures and such... What are your thoughts on the matter?
(fanatics with death threats for trying to add something to kernel,
please abstain)

-Habbit

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
> Hi there everybody,
> 
> I'm opening the RFC because I want to add some kind of infrastructure to
> retrieve the address of system/platform structures. I will explain
> myself: my use case is in i386-pc and for the drivemap module, in which
> a function installs a TSR int13h handler. This requires the function to
> have access to two real mode structures, namely the BIOS Data Area,
> which is based at 0040:0000h; and the Interrupt Vector Table, which
> conventionally starts at 0 but that could have been placed elsewhere by
> the use of the LIDT instruction.

But it is designed to use linear address space for memory so no need to
worry about it. I do not see any reason why there would be paging or
non-linear memory mapping in GRUB 2 (i386-pcbios).

Or did I miss something?

So basically I do not see need for such services. As this does not even
need to be platform independent.

If you need to alter IVT you can modify it on the fly. Though you have
to remember where to use only LOW mem addresses in there.

Re: [RFC] Platform information services

[Javier Martín]

2008/8/14 Vesa Jääskeläinen <chaac@nic.fi>:
> Javier Martín wrote:
>> Hi there everybody,
>>
>> I'm opening the RFC because I want to add some kind of infrastructure to
>> retrieve the address of system/platform structures. I will explain
>> myself: my use case is in i386-pc and for the drivemap module, in which
>> a function installs a TSR int13h handler. This requires the function to
>> have access to two real mode structures, namely the BIOS Data Area,
>> which is based at 0040:0000h; and the Interrupt Vector Table, which
>> conventionally starts at 0 but that could have been placed elsewhere by
>> the use of the LIDT instruction.
>
> But it is designed to use linear address space for memory so no need to
> worry about it. I do not see any reason why there would be paging or
> non-linear memory mapping in GRUB 2 (i386-pcbios).
Yes, but this is a "kernel" design decision that is specified nowhere
to the modules writers. Thus, this could change tomorrow and the
scheme would break down.
>
> Or did I miss something?
>
> So basically I do not see need for such services. As this does not even
> need to be platform independent.
>
> If you need to alter IVT you can modify it on the fly. Though you have
> to remember where to use only LOW mem addresses in there.
Not just low mem addresses, I need to compute the real mode far
pointer (seg:off). Besides, while the BDA always starts at 0040:0000,
the IVT could have been relocated by either the BIOS or GRUB itself,
so I need to use SIDT, which is not a privileged instruction itself
but requires a privileged drop to r-mode to get the IVT address
instead of the pmode IDT.

-Habbit
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
> 2008/8/14 Vesa Jääskeläinen <chaac@nic.fi>:
>> Javier Martín wrote:
>>> Hi there everybody,
>>>
>>> I'm opening the RFC because I want to add some kind of infrastructure to
>>> retrieve the address of system/platform structures. I will explain
>>> myself: my use case is in i386-pc and for the drivemap module, in which
>>> a function installs a TSR int13h handler. This requires the function to
>>> have access to two real mode structures, namely the BIOS Data Area,
>>> which is based at 0040:0000h; and the Interrupt Vector Table, which
>>> conventionally starts at 0 but that could have been placed elsewhere by
>>> the use of the LIDT instruction.
>> But it is designed to use linear address space for memory so no need to
>> worry about it. I do not see any reason why there would be paging or
>> non-linear memory mapping in GRUB 2 (i386-pcbios).
> Yes, but this is a "kernel" design decision that is specified nowhere
> to the modules writers. Thus, this could change tomorrow and the
> scheme would break down.

Still I do not that being changed on GRUB 2 (i386-pcbios). And basically
generic module writers do not need to care, and if it is platform
specific module then you can adapt to that platform if needed. Think it
like device driver.

>> Or did I miss something?
>>
>> So basically I do not see need for such services. As this does not even
>> need to be platform independent.
>>
>> If you need to alter IVT you can modify it on the fly. Though you have
>> to remember where to use only LOW mem addresses in there.
> Not just low mem addresses, I need to compute the real mode far
> pointer (seg:off). Besides, while the BDA always starts at 0040:0000,
> the IVT could have been relocated by either the BIOS or GRUB itself,
> so I need to use SIDT, which is not a privileged instruction itself
> but requires a privileged drop to r-mode to get the IVT address
> instead of the pmode IDT.

Last time I checked seg:off conversion was (seg << 4) | off giving
access to 1 MiB of memory.

BIOS will not relocate this as it is assumed in later stages to be on
that address. And I do not see GRUB doing that as it would need to keep
that table in same place as where BIOS has stored it.

If you still prefer to query it on x86, then there is nothing there to
limit you on doing that. You can always write code to low mem area to
have a real mode call to query any addresses you wish. But I still do
not see any reason why you would write heavy wrapper for that kind of
information especially when those are really platform specific and are
usually not shared between systems. Every platform should use familiar
and convenient method do query that kind of information.

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 4605 bytes --]

El jue, 14-08-2008 a las 20:41 +0300, Vesa Jääskeläinen escribió:
> Javier Martín wrote:
> > 2008/8/14 Vesa Jääskeläinen <chaac@nic.fi>:
> >> Javier Martín wrote:
> >>> Hi there everybody,
> >>>
> >>> I'm opening the RFC because I want to add some kind of infrastructure to
> >>> retrieve the address of system/platform structures. I will explain
> >>> myself: my use case is in i386-pc and for the drivemap module, in which
> >>> a function installs a TSR int13h handler. This requires the function to
> >>> have access to two real mode structures, namely the BIOS Data Area,
> >>> which is based at 0040:0000h; and the Interrupt Vector Table, which
> >>> conventionally starts at 0 but that could have been placed elsewhere by
> >>> the use of the LIDT instruction.
> >> But it is designed to use linear address space for memory so no need to
> >> worry about it. I do not see any reason why there would be paging or
> >> non-linear memory mapping in GRUB 2 (i386-pcbios).
> > Yes, but this is a "kernel" design decision that is specified nowhere
> > to the modules writers. Thus, this could change tomorrow and the
> > scheme would break down.
> 
> Still I do not that being changed on GRUB 2 (i386-pcbios). And basically
> generic module writers do not need to care, and if it is platform
> specific module then you can adapt to that platform if needed. Think it
> like device driver.
Device drivers are just the kind of example that I'm basing upon: the
part related to their hardware they manipulate as directly and
recklessly as the OS kernel allows them, but WRT other parts of the
system they rely on a very defined kernel-driver interface (NDIS,
anyone?) that simply is very nebulous in GRUB.

You say that paging will never be enabled in GRUB2, and it is very
likely that your assertion holds, but what if we suddenly find some kind
of benefit to enable it and some other rock to be walked around requires
that the first MB of memory not be identity mapped? Kaboom for modules
assuming it is: they were assuming things about the implementation of
the kernel that do _not_ have to hold as they are nowhere specified
> 
> >> Or did I miss something?
> >>
> >> So basically I do not see need for such services. As this does not even
> >> need to be platform independent.
> >>
> >> If you need to alter IVT you can modify it on the fly. Though you have
> >> to remember where to use only LOW mem addresses in there.
> > Not just low mem addresses, I need to compute the real mode far
> > pointer (seg:off). Besides, while the BDA always starts at 0040:0000,
> > the IVT could have been relocated by either the BIOS or GRUB itself,
> > so I need to use SIDT, which is not a privileged instruction itself
> > but requires a privileged drop to r-mode to get the IVT address
> > instead of the pmode IDT.
> 
> Last time I checked seg:off conversion was (seg << 4) | off giving
> access to 1 MiB of memory.
I meant that _if_ some kind of mapping was enabled, such conversion
would be non-trivial and possibly only known to the kernel.
> 
> BIOS will not relocate this as it is assumed in later stages to be on
> that address. And I do not see GRUB doing that as it would need to keep
> that table in same place as where BIOS has stored it.
All x86-32/64 BIOSes could perfectly relocate the IVT because they feel
like it, as the LIDT/SIDT operations were introduced with the 80286.
Most current OSes (Linux, Windows from the NT family, etc.) don't care
about the IVT just because they have no need of using it ever again,
start with a cli and jump into pmode as soon as they can. 
> 
> If you still prefer to query it on x86, then there is nothing there to
> limit you on doing that. You can always write code to low mem area to
> have a real mode call to query any addresses you wish. But I still do
> not see any reason why you would write heavy wrapper for that kind of
> information especially when those are really platform specific and are
> usually not shared between systems. Every platform should use familiar
> and convenient method do query that kind of information.
As I showed in my reply to Robert's message, the wrapper need not be
"heavy": most likely the opposite, as a "stub" implementation for
platforms not providing any structures is under 50 bytes, and the
"wrapper" for other platforms consists mainly of a switch-case.

-Habbit
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Robert Millan]

On Thu, Aug 14, 2008 at 06:38:41PM +0200, Javier Martín wrote:
> Yes, but this is a "kernel" design decision that is specified nowhere
> to the modules writers. Thus, this could change tomorrow and the
> scheme would break down.

GRUB is being developed as a whole, which includes kernel and modules.  If
we modify the kernel in a way that could break modules, it's part of the
procedure to check modules and make sure they don't break because of this
(although, of course, we could always make mistakes).

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

El jue, 14-08-2008 a las 20:00 +0200, Robert Millan escribió:
> On Thu, Aug 14, 2008 at 06:38:41PM +0200, Javier Martín wrote:
> > Yes, but this is a "kernel" design decision that is specified nowhere
> > to the modules writers. Thus, this could change tomorrow and the
> > scheme would break down.
> 
> GRUB is being developed as a whole, which includes kernel and modules.  If
> we modify the kernel in a way that could break modules, it's part of the
> procedure to check modules and make sure they don't break because of this
> (although, of course, we could always make mistakes).
> 

That is a very sane policy, but in this case I'm arguing that an
improvement could be made so that such process would not be necessary,
or would be much lighter, regarding a particular section of the kernel
because modules would not rely on certain assumptions about the internal
implementation of GRUB: separation of interface and implementation.
However, there is not a consistent, fully-documented kernel-module
interface, and module developers usually do not know what invariants
hold in their environment - they find it by trial and error. This needs
to change eventually, particularly in the documentation part (please
avoid GRUB ending like ALSA): the GRUBInternals page in the wiki should
be filled up and updated a bit more often.

WRT "kernel and modules going hand by hand", think about external
modules: if the drivemap module is finally rejected for introduction in
GRUB, I will not scrap it, but keep it as a module external to the
official GNU sources and possibly offer it in a web in the form of
patches to the official GRUB2. In this case, changes made to the kernel
would not take into account that module, which would break if I weren't
monitoring this list daily.

Additionally, the cost of this function in platforms which don't have
any structs registered yet, as the function could be a stub like this:

void* grub_machine_get_platform_structure (int stidx)
{
  grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported", stidx);
  return 0;
}

The kernel space taken would most likely be less than 50 bytes. For
i386-pc, it could be like this (also lightweight) function:

void* grub_machine_get_platform_structure (int stidx)
{
  grub_errno = GRUB_ERR_NONE;

  switch (stidx)
  {
  case GRUB_MACHINE_I386_IVT:
    return /* Call to asm function that runs SIDT in real mode */ ;
  case GRUB_MACHINE_I386_BDA:
    return (void*)0x400;
  default:
    grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported",
                stidx);
    return 0;
  }
}

BTW, I've noticed that when using the function, if the result is stored
in "void* p", then the success check cannot only rely on "if (p)",
because 0 is also a legal address (i.e. for the IVT). Thus, the checks
should be like "if (p || grub_errno == GRUB_ERR_NONE)" with the
implementation ensuring that errno is correctly set on success.

-Habbit

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
> WRT "kernel and modules going hand by hand", think about external
> modules: if the drivemap module is finally rejected for introduction in
> GRUB, I will not scrap it, but keep it as a module external to the
> official GNU sources and possibly offer it in a web in the form of
> patches to the official GRUB2. In this case, changes made to the kernel
> would not take into account that module, which would break if I weren't
> monitoring this list daily.

Then it is really your problem ;)

> Additionally, the cost of this function in platforms which don't have
> any structs registered yet, as the function could be a stub like this:
> 
> void* grub_machine_get_platform_structure (int stidx)
> {
>   grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported", stidx);
>   return 0;
> }
> 
> The kernel space taken would most likely be less than 50 bytes. For
> i386-pc, it could be like this (also lightweight) function:
> 
> void* grub_machine_get_platform_structure (int stidx)
> {
>   grub_errno = GRUB_ERR_NONE;
> 
>   switch (stidx)
>   {
>   case GRUB_MACHINE_I386_IVT:
>     return /* Call to asm function that runs SIDT in real mode */ ;
>   case GRUB_MACHINE_I386_BDA:
>     return (void*)0x400;
>   default:
>     grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported",
>                 stidx);
>     return 0;
>   }
> }

And what lets assume couple of extra platforms... how about
x86-32bit-efi and ppc. What do they do?

Implement their own enum entries for those indexes and only use their
own indices...?

Where here we are sharing any code? (if we do not count the name of the
fuction.) Interface is kinda useless if there is no possibility that
no-one is sharing its functionality...

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 3490 bytes --]

El vie, 15-08-2008 a las 19:31 +0300, Vesa Jääskeläinen escribió:
> Javier Martín wrote:
> > WRT "kernel and modules going hand by hand", think about external
> > modules: if the drivemap module is finally rejected for introduction in
> > GRUB, I will not scrap it, but keep it as a module external to the
> > official GNU sources and possibly offer it in a web in the form of
> > patches to the official GRUB2. In this case, changes made to the kernel
> > would not take into account that module, which would break if I weren't
> > monitoring this list daily.
> 
> Then it is really your problem ;)
Indeed, but bitrot is not just the real of external modules: it's
happening right now even within the GRUB trunk as you admit in the
"Build problems on powerpc" thread...

> 
> > Additionally, the cost of this function in platforms which don't have
> > any structs registered yet, as the function could be a stub like this:
> > 
> > void* grub_machine_get_platform_structure (int stidx)
> > {
> >   grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported", stidx);
> >   return 0;
> > }
> > 
> > The kernel space taken would most likely be less than 50 bytes. For
> > i386-pc, it could be like this (also lightweight) function:
> > 
> > void* grub_machine_get_platform_structure (int stidx)
> > {
> >   grub_errno = GRUB_ERR_NONE;
> > 
> >   switch (stidx)
> >   {
> >   case GRUB_MACHINE_I386_IVT:
> >     return /* Call to asm function that runs SIDT in real mode */ ;
> >   case GRUB_MACHINE_I386_BDA:
> >     return (void*)0x400;
> >   default:
> >     grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported",
> >                 stidx);
> >     return 0;
> >   }
> > }
> 
> And what lets assume couple of extra platforms... how about
> x86-32bit-efi and ppc. What do they do?
> 
> Implement their own enum entries for those indexes and only use their
> own indices...?
At first, they would just have the stub which does not recognize any
index, but yes, i386-efi devs could decide that certain
firmware-provided structure (like a video modes info table or such, I
don't know the internals of EFI) might be interesting to a module
they're creating, so they create an index for it and add it to the
version of the function in their platform.

If I had not mentioned it before, the function would be declared in a
cross-platform file, but _implemented_ in platform-specific files, and
the indices would be declared in the platform-specific machine.h.  Thus,
there would not be a "single" indices namespace: structure #1 might be
the IVT in i386-pc, but some devices info table in powerpc-ieee1275.

> 
> Where here we are sharing any code? (if we do not count the name of the
> fuction.) Interface is kinda useless if there is no possibility that
> no-one is sharing its functionality...
The idea is a single function to retrieve the addresses of
firmware-provided/used structures.  This includes the IVT and BDA in
i386-pc, but as I said before it could also be used by other platforms
for their own structures.  The alternative would be just creating such
"get struct X" functions on each platform as they are needed, but I
imagined that a single interface (with such a low cost in space) would
be a more elegant solution.

-Habbit
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
> El vie, 15-08-2008 a las 19:31 +0300, Vesa Jääskeläinen escribió:
>> Javier Martín wrote:
>>> WRT "kernel and modules going hand by hand", think about external
>>> modules: if the drivemap module is finally rejected for introduction in
>>> GRUB, I will not scrap it, but keep it as a module external to the
>>> official GNU sources and possibly offer it in a web in the form of
>>> patches to the official GRUB2. In this case, changes made to the kernel
>>> would not take into account that module, which would break if I weren't
>>> monitoring this list daily.
>> Then it is really your problem ;)
> Indeed, but bitrot is not just the real of external modules: it's
> happening right now even within the GRUB trunk as you admit in the
> "Build problems on powerpc" thread...

And? If Power PC maintainer is nowhere to update to newest additions
then it is rightly in in-compilable state and if it rots too long its
support will be removed. That's life.

>>> Additionally, the cost of this function in platforms which don't have
>>> any structs registered yet, as the function could be a stub like this:
>>>
>>> void* grub_machine_get_platform_structure (int stidx)
>>> {
>>>   grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported", stidx);
>>>   return 0;
>>> }
>>>
>>> The kernel space taken would most likely be less than 50 bytes. For
>>> i386-pc, it could be like this (also lightweight) function:
>>>
>>> void* grub_machine_get_platform_structure (int stidx)
>>> {
>>>   grub_errno = GRUB_ERR_NONE;
>>>
>>>   switch (stidx)
>>>   {
>>>   case GRUB_MACHINE_I386_IVT:
>>>     return /* Call to asm function that runs SIDT in real mode */ ;
>>>   case GRUB_MACHINE_I386_BDA:
>>>     return (void*)0x400;
>>>   default:
>>>     grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported",
>>>                 stidx);
>>>     return 0;
>>>   }
>>> }
>> And what lets assume couple of extra platforms... how about
>> x86-32bit-efi and ppc. What do they do?
>>
>> Implement their own enum entries for those indexes and only use their
>> own indices...?
> At first, they would just have the stub which does not recognize any
> index, but yes, i386-efi devs could decide that certain
> firmware-provided structure (like a video modes info table or such, I
> don't know the internals of EFI) might be interesting to a module
> they're creating, so they create an index for it and add it to the
> version of the function in their platform.
> 
> If I had not mentioned it before, the function would be declared in a
> cross-platform file, but _implemented_ in platform-specific files, and
> the indices would be declared in the platform-specific machine.h.  Thus,
> there would not be a "single" indices namespace: structure #1 might be
> the IVT in i386-pc, but some devices info table in powerpc-ieee1275.
> 
>> Where here we are sharing any code? (if we do not count the name of the
>> fuction.) Interface is kinda useless if there is no possibility that
>> no-one is sharing its functionality...
> The idea is a single function to retrieve the addresses of
> firmware-provided/used structures.  This includes the IVT and BDA in
> i386-pc, but as I said before it could also be used by other platforms
> for their own structures.  The alternative would be just creating such
> "get struct X" functions on each platform as they are needed, but I
> imagined that a single interface (with such a low cost in space) would
> be a more elegant solution.

I still do not see the need to create cross platform function that
cannot be used for cross platform purposes. It is much more reasonable
to write such needs as in kernel for the platform or in platform
specific modules.

I do not have a problem with function to retrieve pointer to some
platform specific function on platfrom specific code. That is normal
life of the platform.

And if we think about code safety, casting is bad. And if you keep
indices colliding on different platforms then you are just calling for
problems. It completely removes any advantage what this kinda wrapper
could have had.

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 5374 bytes --]

El vie, 15-08-2008 a las 20:14 +0300, Vesa Jääskeläinen escribió:
> Javier Martín wrote:
> > El vie, 15-08-2008 a las 19:31 +0300, Vesa Jääskeläinen escribió:
> >> Javier Martín wrote:
> >>> WRT "kernel and modules going hand by hand", think about external
> >>> modules: if the drivemap module is finally rejected for introduction in
> >>> GRUB, I will not scrap it, but keep it as a module external to the
> >>> official GNU sources and possibly offer it in a web in the form of
> >>> patches to the official GRUB2. In this case, changes made to the kernel
> >>> would not take into account that module, which would break if I weren't
> >>> monitoring this list daily.
> >> Then it is really your problem ;)
> > Indeed, but bitrot is not just the real of external modules: it's
> > happening right now even within the GRUB trunk as you admit in the
> > "Build problems on powerpc" thread...
> 
> And? If Power PC maintainer is nowhere to update to newest additions
> then it is rightly in in-compilable state and if it rots too long its
> support will be removed. That's life.
Yes, but you implicitly and particularly Robert explicitly argued that
when kernel changes devs here take the time to update modules in
consonance and vice versa.  The PPC build problem shows that, for
whatever reason, this is not always true.
> 
> >>> Additionally, the cost of this function in platforms which don't have
> >>> any structs registered yet, as the function could be a stub like this:
> >>>
> >>> void* grub_machine_get_platform_structure (int stidx)
> >>> {
> >>>   grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported", stidx);
> >>>   return 0;
> >>> }
> >>>
> >>> The kernel space taken would most likely be less than 50 bytes. For
> >>> i386-pc, it could be like this (also lightweight) function:
> >>>
> >>> void* grub_machine_get_platform_structure (int stidx)
> >>> {
> >>>   grub_errno = GRUB_ERR_NONE;
> >>>
> >>>   switch (stidx)
> >>>   {
> >>>   case GRUB_MACHINE_I386_IVT:
> >>>     return /* Call to asm function that runs SIDT in real mode */ ;
> >>>   case GRUB_MACHINE_I386_BDA:
> >>>     return (void*)0x400;
> >>>   default:
> >>>     grub_error (GRUB_ERR_BAD_ARGUMENT, "Struct %d not supported",
> >>>                 stidx);
> >>>     return 0;
> >>>   }
> >>> }
> >> And what lets assume couple of extra platforms... how about
> >> x86-32bit-efi and ppc. What do they do?
> >>
> >> Implement their own enum entries for those indexes and only use their
> >> own indices...?
> > At first, they would just have the stub which does not recognize any
> > index, but yes, i386-efi devs could decide that certain
> > firmware-provided structure (like a video modes info table or such, I
> > don't know the internals of EFI) might be interesting to a module
> > they're creating, so they create an index for it and add it to the
> > version of the function in their platform.
> > 
> > If I had not mentioned it before, the function would be declared in a
> > cross-platform file, but _implemented_ in platform-specific files, and
> > the indices would be declared in the platform-specific machine.h.  Thus,
> > there would not be a "single" indices namespace: structure #1 might be
> > the IVT in i386-pc, but some devices info table in powerpc-ieee1275.
> > 
> >> Where here we are sharing any code? (if we do not count the name of the
> >> fuction.) Interface is kinda useless if there is no possibility that
> >> no-one is sharing its functionality...
> > The idea is a single function to retrieve the addresses of
> > firmware-provided/used structures.  This includes the IVT and BDA in
> > i386-pc, but as I said before it could also be used by other platforms
> > for their own structures.  The alternative would be just creating such
> > "get struct X" functions on each platform as they are needed, but I
> > imagined that a single interface (with such a low cost in space) would
> > be a more elegant solution.
> 
> I still do not see the need to create cross platform function that
> cannot be used for cross platform purposes. It is much more reasonable
> to write such needs as in kernel for the platform or in platform
> specific modules.
> 
> I do not have a problem with function to retrieve pointer to some
> platform specific function on platfrom specific code. That is normal
> life of the platform.
Well, think of this as a platform-specific function to retrieve the
address of platform-specific (firmware-provided) structures. The only
difference is that in my proposal such a function has the same signature
(prototype) for every platform.
> 
> And if we think about code safety, casting is bad. And if you keep
> indices colliding on different platforms then you are just calling for
> problems. It completely removes any advantage what this kinda wrapper
> could have had.
Why? any module that is "advanced" (in the sense of complicated) enough
to require the use of platform-specific structures will be
platform-specific itself, or at least the part of it that accesses the
structures, and thus the risk of indices actually colliding is nearly
zero.

-Habbit
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
>> I still do not see the need to create cross platform function that
>> cannot be used for cross platform purposes. It is much more reasonable
>> to write such needs as in kernel for the platform or in platform
>> specific modules.
>>
>> I do not have a problem with function to retrieve pointer to some
>> platform specific function on platfrom specific code. That is normal
>> life of the platform.
> Well, think of this as a platform-specific function to retrieve the
> address of platform-specific (firmware-provided) structures. The only
> difference is that in my proposal such a function has the same signature
> (prototype) for every platform.

Well... THEN WHY NOT MAKE A PLATFORM SPECIFIC FUCTION THAT IS CONVENIENT
TO USE ON PLATFORM WHERE IT IS MEANT TO BE USED FOR AND IT ACTUALLY HAVE
A MEANING ON WHERE IT IS ACCESSIBLE.

If you want to learn something, study interfaces of disk->biosdisk->bios
and video->vbe->video bios... There are also several other good spots on
GRUB 2 code base that you could see...

You have still not provided a single GOOD example where this would be
used AND would provide some ADDED VALUE. Unless you provide a convincing
one (while I pretty much doubt) I do not see this feature to be in our
official tree.

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 4174 bytes --]

El vie, 15-08-2008 a las 21:35 +0300, Vesa Jääskeläinen escribió:
> Javier Martín wrote:
> >> I still do not see the need to create cross platform function that
> >> cannot be used for cross platform purposes. It is much more reasonable
> >> to write such needs as in kernel for the platform or in platform
> >> specific modules.
> >>
> >> I do not have a problem with function to retrieve pointer to some
> >> platform specific function on platfrom specific code. That is normal
> >> life of the platform.
> > Well, think of this as a platform-specific function to retrieve the
> > address of platform-specific (firmware-provided) structures. The only
> > difference is that in my proposal such a function has the same signature
> > (prototype) for every platform.
> 
> Well... THEN WHY NOT MAKE A PLATFORM SPECIFIC FUCTION THAT IS CONVENIENT
> TO USE ON PLATFORM WHERE IT IS MEANT TO BE USED FOR AND IT ACTUALLY HAVE
> A MEANING ON WHERE IT IS ACCESSIBLE.
I take that by "convenient to use" and your earlier "code safety"
reference, you imply a function that returns a typed pointer to the
structure, like "bda* var = grub_getbda();" in i386-pc, so that fields
of such struct can be directly accessed by var->free_mem_kb.

However, this requires that a function is created for each firmware
structure that is deemed interesting by GRUB developers. In this
proposal, only one (very lightweight) function is created and then IDs
are defined for the structs. WRT "code safety", malloc and friends also
return a void* that you have to cast, so nothing new under the sky.

> 
> If you want to learn something, study interfaces of disk->biosdisk->bios
> and video->vbe->video bios... There are also several other good spots on
> GRUB 2 code base that you could see...
Will do, particularly the biosdisk part (video code nearly always makes
my head spin).

> 
> You have still not provided a single GOOD example where this would be
> used AND would provide some ADDED VALUE. Unless you provide a convincing
> one (while I pretty much doubt) I do not see this feature to be in our
> official tree.
The added value (that expression reminds me of taxes) is:
 -Compared to SVN HEAD: a way to retrieve the _module-accessible_
address of firmware-provided structures. This is a protection against
future changes in the GRUB kernel (like enabling paging, changing memory
mapping schemes, etc.) that would otherwise force to inspect every
single module for accesses to such structures by their "real" addresses.
 -Compared to SVN HEAD plus the many ad-hoc functions you seem to
support: for most of the structures an asm helper is not required, thus
less kernel space overhead than with many functions. Also, "focus of
efforts" - maybe the oversight on "the" core function would add
additional bitrot protection compared to the individuals functions.

An use case would be in the proposed drivemap module, in the function
installing an interrupt handler in the real-mode IVT: instead of the
current code with magic addresses,

/* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
grub_uint32_t *ivtslot = (grub_uint32_t*)0x0000004c;

The module could be like this:

/* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
grub_uint32_t *ivtslot = 19 +
(grub_uint32_t*)grub_machine_get_fwstruct(GRUB_I386_PC_IVT);

Then, if anything of the following ever happens:

   * The real mode IVT was live-relocated, either by GRUB or by the
BIOS, by the use of LIDT
   * The real mode IVT was destroyed by GRUB, which keeps a copy at
another address and sets it up, with the rest of a 8086-compatible
environment only when it is required (memory pressure maybe?).
   * Linear addresses 0-1M do not correspond with physical addresses
0-1M because of whatever memory management the GRUB kernel is doing

The above code would continue to work, while the even-more-above code
would fail with unpredictable results.

-Habbit
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
> I take that by "convenient to use" and your earlier "code safety"
> reference, you imply a function that returns a typed pointer to the
> structure, like "bda* var = grub_getbda();" in i386-pc, so that fields
> of such struct can be directly accessed by var->free_mem_kb.
> 
> However, this requires that a function is created for each firmware
> structure that is deemed interesting by GRUB developers. In this
> proposal, only one (very lightweight) function is created and then IDs
> are defined for the structs. WRT "code safety", malloc and friends also
> return a void* that you have to cast, so nothing new under the sky.

In your case there is really no change.

You create new switch branch (which is most likely implemented by
function somewhere).

In order to access this void * you have to know its contents. Why not
use structure to define its contents where viable?

When using malloc() you know that memory returned is not formatted in
anyway. Therefore it is safe to cast.

You just want to make it more complex that it needs to be in my eyes.

>> You have still not provided a single GOOD example where this would be
>> used AND would provide some ADDED VALUE. Unless you provide a convincing
>> one (while I pretty much doubt) I do not see this feature to be in our
>> official tree.
> An use case would be in the proposed drivemap module, in the function
> installing an interrupt handler in the real-mode IVT: instead of the
> current code with magic addresses,
> 
> /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
> grub_uint32_t *ivtslot = (grub_uint32_t*)0x0000004c;
> 
> The module could be like this:
> 
> /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
> grub_uint32_t *ivtslot = 19 +
> (grub_uint32_t*)grub_machine_get_fwstruct(GRUB_I386_PC_IVT);

The better way that complies with all your error cases provided:

grub_ivt_entry_t *entry;
entry = grub_get_ivt_entry(0x13);
*entry = new_value;

of course grub_ivt_entry_t could be renamed to be something like
grub_realmode_addr_t.

.

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2919 bytes --]

El vie, 15-08-2008 a las 22:46 +0300, Vesa Jääskeläinen escribió:
> Javier Martín wrote:
> > I take that by "convenient to use" and your earlier "code safety"
> > reference, you imply a function that returns a typed pointer to the
> > structure, like "bda* var = grub_getbda();" in i386-pc, so that fields
> > of such struct can be directly accessed by var->free_mem_kb.
> > 
> > However, this requires that a function is created for each firmware
> > structure that is deemed interesting by GRUB developers. In this
> > proposal, only one (very lightweight) function is created and then IDs
> > are defined for the structs. WRT "code safety", malloc and friends also
> > return a void* that you have to cast, so nothing new under the sky.
> 
> In your case there is really no change.
> 
> You create new switch branch (which is most likely implemented by
> function somewhere).
> 
> In order to access this void * you have to know its contents. Why not
> use structure to define its contents where viable?
True, I just wanted to avoid defining structures that would only be used
once or twice, but you're right.
> 
> When using malloc() you know that memory returned is not formatted in
> anyway. Therefore it is safe to cast.
> 
> You just want to make it more complex that it needs to be in my eyes.
> 
> >> You have still not provided a single GOOD example where this would be
> >> used AND would provide some ADDED VALUE. Unless you provide a convincing
> >> one (while I pretty much doubt) I do not see this feature to be in our
> >> official tree.
> > An use case would be in the proposed drivemap module, in the function
> > installing an interrupt handler in the real-mode IVT: instead of the
> > current code with magic addresses,
> > 
> > /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
> > grub_uint32_t *ivtslot = (grub_uint32_t*)0x0000004c;
> > 
> > The module could be like this:
> > 
> > /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
> > grub_uint32_t *ivtslot = 19 +
> > (grub_uint32_t*)grub_machine_get_fwstruct(GRUB_I386_PC_IVT);
> 
> The better way that complies with all your error cases provided:
> 
> grub_ivt_entry_t *entry;
> entry = grub_get_ivt_entry(0x13);
> *entry = new_value;
This was another of the solutions I was considering in this thread (the
"many-functions approach"), I just found the single-function solution
more elegant in the case of "large numbers" (>10) of structures becoming
visible through this method, but your snippet looks fine and "safer".

> 
> of course grub_ivt_entry_t could be renamed to be something like
> grub_realmode_addr_t.
grub_realmode_farptr_t would suit better IMO.

-Habbit
> 
> .
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Javier Martín wrote:
>>> An use case would be in the proposed drivemap module, in the function
>>> installing an interrupt handler in the real-mode IVT: instead of the
>>> current code with magic addresses,
>>>
>>> /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
>>> grub_uint32_t *ivtslot = (grub_uint32_t*)0x0000004c;
>>>
>>> The module could be like this:
>>>
>>> /* Real mode IVT slot (seg:off far pointer) for interrupt 0x13.  */
>>> grub_uint32_t *ivtslot = 19 +
>>> (grub_uint32_t*)grub_machine_get_fwstruct(GRUB_I386_PC_IVT);
>> The better way that complies with all your error cases provided:
>>
>> grub_ivt_entry_t *entry;
>> entry = grub_get_ivt_entry(0x13);
>> *entry = new_value;
> This was another of the solutions I was considering in this thread (the
> "many-functions approach"), I just found the single-function solution
> more elegant in the case of "large numbers" (>10) of structures becoming
> visible through this method, but your snippet looks fine and "safer".

It is only visible within platform where it is used, so that is not a
problem. "Large number" of structures are still needed for that platform
(and if they are not needed, then there is no need for the function) and
that need cannot be discarded in anyway. Unless you want to use magic
numbers to define offsets...

So I would dismiss the idea and continue to use other methods.

>> of course grub_ivt_entry_t could be renamed to be something like
>> grub_realmode_addr_t.
> grub_realmode_farptr_t would suit better IMO.

Please be free to use that.

Re: [RFC] Platform information services

[Isaac Dupree]

my view of the "interface breakage" problem in this project:

Either the property of the kernel you're relying on is going 
to change, or it isn't; it doesn't/shouldn't have stable 
APIs, and it will change only if there's a reason for it to 
change.  So I think the best we can do is to document what 
code relies on what properties of the kernel.  Wherever's 
appropriate and will likely be noticed and kept up-to-date. 
for example:

if you rely on a property of the kernel, you should document 
it somewhere in the kernel, and document everyone 
(hopefully) who relies on that property; or the users could 
have some symbol in the source that you can search for.  It 
doesn't have to be a function that takes up any actual 
kernel disk-space.

a bit at a distance,
-Isaac

Re: [RFC] Platform information services

[Javier Martín]

[-- Attachment #1: Type: text/plain, Size: 2207 bytes --]

El vie, 15-08-2008 a las 18:38 -0400, Isaac Dupree escribió:
> my view of the "interface breakage" problem in this project:
> 
> Either the property of the kernel you're relying on is going 
> to change, or it isn't; it doesn't/shouldn't have stable 
> APIs, and it will change only if there's a reason for it to 
> change.  So I think the best we can do is to document what 
> code relies on what properties of the kernel.  Wherever's 
> appropriate and will likely be noticed and kept up-to-date. 
> for example:
> 
> if you rely on a property of the kernel, you should document 
> it somewhere in the kernel, and document everyone 
> (hopefully) who relies on that property; or the users could 
> have some symbol in the source that you can search for.  It 
> doesn't have to be a function that takes up any actual 
> kernel disk-space.
> 
> a bit at a distance,
> -Isaac
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

The problem is exactly that: for the sake of modularity we are writing
module code that requires the manipulation of firmware-provided
structures, but we are relying on kernel implementation details that are
nowhere specified in the kernel-module interfaces like "the GRUB kernel
will not activate any kind of paging and in any event the 0-1M+64K
memory region will always be identity-mapped". Even when some parts
_might_ be documented in the wiki (like the MemoryMap page), other parts
of the wiki are so outdated and wrong that one cannot trust the good
pages! Besides, does the info in MemoryMap describe the layout of the
physical address space or the linear address space that modules see? etc

Thus, either we create interfaces in the kernel to deal with the
possible breakage of these assumptions (i.e. the platform info
function<s> we've been discussing in this thread would take care of
translating addresses across any memory mappings) or we set those
assumptions in stone and turn them into part of the kernel-module
interface, _documenting them_. Seriously, InteralsIntro et al. need a
lot more love...

-Habbit

[-- Attachment #2: Esta parte del mensaje está firmada digitalmente --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [RFC] Platform information services

[Vesa Jääskeläinen]

Isaac Dupree wrote:
> if you rely on a property of the kernel, you should document it
> somewhere in the kernel, and document everyone (hopefully) who relies on
> that property; or the users could have some symbol in the source that
> you can search for.  It doesn't have to be a function that takes up any
> actual kernel disk-space.

Function name is kinda good symbol. Magic numbers without defines are
not used really on GRUB 2 code base, if they are then there is omission
and that should be fixed. Every good editor allows you to allow fast
walking on source code based on symbols.

But I am against documenting into kernel who use what. It is doomed to
be outdated easily and that information is easily available when using
decent editor.

Of course you should document what code does.

Re: [RFC] Platform information services

[Robert Millan]

On Fri, Aug 15, 2008 at 07:03:18PM +0200, Javier Martín wrote:
> El vie, 15-08-2008 a las 19:31 +0300, Vesa Jääskeläinen escribió:
> > Javier Martín wrote:
> > > WRT "kernel and modules going hand by hand", think about external
> > > modules: if the drivemap module is finally rejected for introduction in
> > > GRUB, I will not scrap it, but keep it as a module external to the
> > > official GNU sources and possibly offer it in a web in the form of
> > > patches to the official GRUB2. In this case, changes made to the kernel
> > > would not take into account that module, which would break if I weren't
> > > monitoring this list daily.
> > 
> > Then it is really your problem ;)
> Indeed, but bitrot is not just the real of external modules: it's
> happening right now even within the GRUB trunk

We shouldn't compromise GRUB design to work around maintainance problems.  We
don't make any ABI or even API promises, except for Multiboot.  I don't see
why we need this burden simply because some patches take time to be merged.

> as you admit in the
> "Build problems on powerpc" thread...

The problem with the powerpc port is that noone is actively maintaining it.  It
has nothing to do with design decisions.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."