Exposing device ids and driver names

Exposing device ids and driver names

[Nathaniel McCallum]

Please CC me on any responses as I'm not subscribed to lkml.

I have the aim at creating two tools helpful to linux.  The first tool 
is a driver regression test of sorts.  I want to be able to create 
essentially a time line of hardware support as they appear in distros. 
The second tool, related to the first, is a program which runs on 
Windows and scans for a user's hardware and tells them which distro will 
best support their hardware.  I already have a working prototype of 
these two tools.  It currently uses the data exported by modinfo.  This 
however does not provide transparency for drivers compiled into the kernel.

Thus, my requirements are as follows.  I need to be able to gather the 
*_device_id tables and driver names from the kernel by scanning the 
kernel/module binaries.  The implementation should be careful not to 
expand memory footprint if at all possible and should seek to expose 
only stable structures to userspace.

Further, I have the limitation that symbol names are unhelpful since 
they are stripped out of the kernel.

My initial investigation has led me to one of two methods (though I am 
certainly open to other suggestions).  Since symbols are most often 
stripped from the kernel, I have largely focused my investigation on 
using ELF sections.

The first method is to expand the MODULE_DEVICE_TABLE() macro to do 
something for built in modules (such as the following):
#define MODULE_DEVICE_TABLE(type, name)                        \
   static const struct type##_device_id *__##name##_dev_tbl     \
     __used __section(.dev.ids.##type)

This has the advantage of being an extremely small change to the kernel 
and being easy to parse out of the binaries.  The disadvantages are that 
it adds a pointer for each device table (expanding memory footprint) and 
does not provide the name of the driver which will be loaded for that 
table (making it hard to track when a driver moves in/out of the kernel 
to/from a module).

The second method is to create a macro for *_driver struct definitions 
(similar to DEFINE_PCI_DEVICE_TABLE() in include/linux/pci.h).  Perhaps 
something like this:
#define DEFINE_DRIVER(type, name) \
   static struct type##_driver name __section(.drv.##type)

The advantages of this method are minimal memory footprint increase 
(only the size of the ELF section header and name) and easy parsing of 
both the driver name and the type##_device_id struct.  The disadvantages 
are that every driver will have to be updated to use the new macro 
(which can probably be done programatically) and that *_driver structs 
have now become part of the kernel ABI (I'm not sure how bad this is). 
I also ran into a wierd problem where the *_driver structs are zeroed 
when built into modules, but I'm not sure what causes that (it wasn't my 
patch).

Which method is preferable? Any other suggestions?

Thanks for your help,
Nathaniel

Re: Exposing device ids and driver names

[Greg KH]

On Thu, Oct 01, 2009 at 12:40:05PM -0400, Nathaniel McCallum wrote:
> Please CC me on any responses as I'm not subscribed to lkml.
> 
> I have the aim at creating two tools helpful to linux.  The first tool 
> is a driver regression test of sorts.  I want to be able to create 
> essentially a time line of hardware support as they appear in distros. 
> The second tool, related to the first, is a program which runs on 
> Windows and scans for a user's hardware and tells them which distro will 
> best support their hardware.

That's going to be interesting, as all distros pretty much use the same
kernel, it will just depend on who is "newer" at the moment, right?

> I already have a working prototype of these two tools.  It currently
> uses the data exported by modinfo.  This however does not provide
> transparency for drivers compiled into the kernel.

Most distros don't build drivers into the kernel, so you should be fine
with what you have today, right?  Or have you run into problems with
this?

Why not just use the baseline kernel as a model for this.  Do a 'make
allmodconfig' and then extract the data and publish it that way.  No
kernel changes are needed, and then any distro can be easily matched up
by this based on what they are using.  That will save you time in
downloading zillions of distro releases, and provide a nice easy way to
show what the kernel.org releases support.

thanks,

greg k-h

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 12:42 PM, Greg KH wrote:
> On Thu, Oct 01, 2009 at 12:40:05PM -0400, Nathaniel McCallum wrote:
>> Please CC me on any responses as I'm not subscribed to lkml.
>>
>> I have the aim at creating two tools helpful to linux.  The first tool
>> is a driver regression test of sorts.  I want to be able to create
>> essentially a time line of hardware support as they appear in distros.
>> The second tool, related to the first, is a program which runs on
>> Windows and scans for a user's hardware and tells them which distro will
>> best support their hardware.
>
> That's going to be interesting, as all distros pretty much use the same
> kernel, it will just depend on who is "newer" at the moment, right?

Mostly yes.  I intend to also trace id diffs against mainline to see if 
distros have picked up patches that are not upstream.

>> I already have a working prototype of these two tools.  It currently
>> uses the data exported by modinfo.  This however does not provide
>> transparency for drivers compiled into the kernel.
>
> Most distros don't build drivers into the kernel, so you should be fine
> with what you have today, right?  Or have you run into problems with
> this?

I've run into specific problems.  This is obviously the case on 
specialized distros (which I *do* want to measure).  However, even a 
quick diff on configs will show differences on major distros with some 
devices being compiled into the kernel.  It is also often true that ACPI 
devices are built into the kernel.

> Why not just use the baseline kernel as a model for this.  Do a 'make
> allmodconfig' and then extract the data and publish it that way.  No
> kernel changes are needed, and then any distro can be easily matched up
> by this based on what they are using.  That will save you time in
> downloading zillions of distro releases, and provide a nice easy way to
> show what the kernel.org releases support.

Unfortunately, I would not be able to track changes to the kernel in 
this model.  Since this is one of my explicit goals (to make sure that 
distro kernel changes get upstream), I think a non-invasive kernel 
modification would be worth the effort.

Nathaniel

Re: Exposing device ids and driver names

[Greg KH]

On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
> On 10/01/2009 12:42 PM, Greg KH wrote:
> > On Thu, Oct 01, 2009 at 12:40:05PM -0400, Nathaniel McCallum wrote:
> >> Please CC me on any responses as I'm not subscribed to lkml.
> >>
> >> I have the aim at creating two tools helpful to linux.  The first tool
> >> is a driver regression test of sorts.  I want to be able to create
> >> essentially a time line of hardware support as they appear in distros.
> >> The second tool, related to the first, is a program which runs on
> >> Windows and scans for a user's hardware and tells them which distro will
> >> best support their hardware.
> >
> > That's going to be interesting, as all distros pretty much use the same
> > kernel, it will just depend on who is "newer" at the moment, right?
> 
> Mostly yes.  I intend to also trace id diffs against mainline to see if 
> distros have picked up patches that are not upstream.

That should be easier to do by just looking at the diff from mainline to
their kernel.  All distros should be providing this already so it
shouldn't be that hard to run a static code analyzer on it if needed.

> >> I already have a working prototype of these two tools.  It currently
> >> uses the data exported by modinfo.  This however does not provide
> >> transparency for drivers compiled into the kernel.
> >
> > Most distros don't build drivers into the kernel, so you should be fine
> > with what you have today, right?  Or have you run into problems with
> > this?
> 
> I've run into specific problems.  This is obviously the case on 
> specialized distros (which I *do* want to measure).  However, even a 
> quick diff on configs will show differences on major distros with some 
> devices being compiled into the kernel.  It is also often true that ACPI 
> devices are built into the kernel.

Yes, and look at the Moblin builds, we put lots of stuff into the kernel
directly for speed at boot issues.

> > Why not just use the baseline kernel as a model for this.  Do a 'make
> > allmodconfig' and then extract the data and publish it that way.  No
> > kernel changes are needed, and then any distro can be easily matched up
> > by this based on what they are using.  That will save you time in
> > downloading zillions of distro releases, and provide a nice easy way to
> > show what the kernel.org releases support.
> 
> Unfortunately, I would not be able to track changes to the kernel in 
> this model.  Since this is one of my explicit goals (to make sure that 
> distro kernel changes get upstream), I think a non-invasive kernel 
> modification would be worth the effort.

But this was an invasive modification, it added space to the kernel
images for no real benifit other than for your tracking tools.  That's
not going to fly unless you can find another good use for the change.

thanks,

greg k-h

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 02:05 PM, Greg KH wrote:
> On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
>> On 10/01/2009 12:42 PM, Greg KH wrote:
>>> Why not just use the baseline kernel as a model for this.  Do a 'make
>>> allmodconfig' and then extract the data and publish it that way.  No
>>> kernel changes are needed, and then any distro can be easily matched up
>>> by this based on what they are using.  That will save you time in
>>> downloading zillions of distro releases, and provide a nice easy way to
>>> show what the kernel.org releases support.
>>
>> Unfortunately, I would not be able to track changes to the kernel in
>> this model.  Since this is one of my explicit goals (to make sure that
>> distro kernel changes get upstream), I think a non-invasive kernel
>> modification would be worth the effort.
>
> But this was an invasive modification, it added space to the kernel
> images for no real benifit other than for your tracking tools.  That's
> not going to fly unless you can find another good use for the change.

Which is why I asked for advice for better options.  I would prefer a 
non-invasive modification.  I am hoping that someone more familiar with 
the layer would provide such a suggestion.

One potential benefit for moving module info to ELF sections would be 
the ability to strip kernel modules.  As a test, I did this on a recent 
Fedora rawhide kernel I had lying around.  Stripping the modules results 
in a 43% decrease in size (82M to 47M).

But I still would prefer a non-invasive solution.

Nathaniel

Re: Exposing device ids and driver names

[Greg KH]

On Thu, Oct 01, 2009 at 02:35:40PM -0400, Nathaniel McCallum wrote:
> On 10/01/2009 02:05 PM, Greg KH wrote:
> > On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
> >> On 10/01/2009 12:42 PM, Greg KH wrote:
> >>> Why not just use the baseline kernel as a model for this.  Do a 'make
> >>> allmodconfig' and then extract the data and publish it that way.  No
> >>> kernel changes are needed, and then any distro can be easily matched up
> >>> by this based on what they are using.  That will save you time in
> >>> downloading zillions of distro releases, and provide a nice easy way to
> >>> show what the kernel.org releases support.
> >>
> >> Unfortunately, I would not be able to track changes to the kernel in
> >> this model.  Since this is one of my explicit goals (to make sure that
> >> distro kernel changes get upstream), I think a non-invasive kernel
> >> modification would be worth the effort.
> >
> > But this was an invasive modification, it added space to the kernel
> > images for no real benifit other than for your tracking tools.  That's
> > not going to fly unless you can find another good use for the change.
> 
> Which is why I asked for advice for better options.  I would prefer a 
> non-invasive modification.  I am hoping that someone more familiar with 
> the layer would provide such a suggestion.
> 
> One potential benefit for moving module info to ELF sections would be 
> the ability to strip kernel modules.  As a test, I did this on a recent 
> Fedora rawhide kernel I had lying around.  Stripping the modules results 
> in a 43% decrease in size (82M to 47M).

Did those modules have debugging symbols enabled?  That seems like a
large savings for just the module device tables.

> But I still would prefer a non-invasive solution.

Agreed, sorry I don't have an idea on how to do it.

thanks,

greg k-h

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 02:40 PM, Greg KH wrote:
> On Thu, Oct 01, 2009 at 02:35:40PM -0400, Nathaniel McCallum wrote:
>> On 10/01/2009 02:05 PM, Greg KH wrote:
>>> On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
>>>> On 10/01/2009 12:42 PM, Greg KH wrote:
>>>>> Why not just use the baseline kernel as a model for this.  Do a 'make
>>>>> allmodconfig' and then extract the data and publish it that way.  No
>>>>> kernel changes are needed, and then any distro can be easily matched up
>>>>> by this based on what they are using.  That will save you time in
>>>>> downloading zillions of distro releases, and provide a nice easy way to
>>>>> show what the kernel.org releases support.
>>>>
>>>> Unfortunately, I would not be able to track changes to the kernel in
>>>> this model.  Since this is one of my explicit goals (to make sure that
>>>> distro kernel changes get upstream), I think a non-invasive kernel
>>>> modification would be worth the effort.
>>>
>>> But this was an invasive modification, it added space to the kernel
>>> images for no real benifit other than for your tracking tools.  That's
>>> not going to fly unless you can find another good use for the change.
>>
>> Which is why I asked for advice for better options.  I would prefer a
>> non-invasive modification.  I am hoping that someone more familiar with
>> the layer would provide such a suggestion.
>>
>> One potential benefit for moving module info to ELF sections would be
>> the ability to strip kernel modules.  As a test, I did this on a recent
>> Fedora rawhide kernel I had lying around.  Stripping the modules results
>> in a 43% decrease in size (82M to 47M).
>
> Did those modules have debugging symbols enabled?  That seems like a
> large savings for just the module device tables.

It does not appear so (none of the debug sections are present).  But I 
could be wrong.

Stripping the modules on the penultimate Fedora 11 kernel results in the 
same drop in size.  I can't imagine why a release kernel would have 
anything extra left in the modules (unless it is just by accident).

Nathaniel

Re: Exposing device ids and driver names

[Greg KH]

On Thu, Oct 01, 2009 at 02:56:55PM -0400, Nathaniel McCallum wrote:
> On 10/01/2009 02:40 PM, Greg KH wrote:
> > On Thu, Oct 01, 2009 at 02:35:40PM -0400, Nathaniel McCallum wrote:
> >> On 10/01/2009 02:05 PM, Greg KH wrote:
> >>> On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
> >>>> On 10/01/2009 12:42 PM, Greg KH wrote:
> >>>>> Why not just use the baseline kernel as a model for this.  Do a 'make
> >>>>> allmodconfig' and then extract the data and publish it that way.  No
> >>>>> kernel changes are needed, and then any distro can be easily matched up
> >>>>> by this based on what they are using.  That will save you time in
> >>>>> downloading zillions of distro releases, and provide a nice easy way to
> >>>>> show what the kernel.org releases support.
> >>>>
> >>>> Unfortunately, I would not be able to track changes to the kernel in
> >>>> this model.  Since this is one of my explicit goals (to make sure that
> >>>> distro kernel changes get upstream), I think a non-invasive kernel
> >>>> modification would be worth the effort.
> >>>
> >>> But this was an invasive modification, it added space to the kernel
> >>> images for no real benifit other than for your tracking tools.  That's
> >>> not going to fly unless you can find another good use for the change.
> >>
> >> Which is why I asked for advice for better options.  I would prefer a
> >> non-invasive modification.  I am hoping that someone more familiar with
> >> the layer would provide such a suggestion.
> >>
> >> One potential benefit for moving module info to ELF sections would be
> >> the ability to strip kernel modules.  As a test, I did this on a recent
> >> Fedora rawhide kernel I had lying around.  Stripping the modules results
> >> in a 43% decrease in size (82M to 47M).
> >
> > Did those modules have debugging symbols enabled?  That seems like a
> > large savings for just the module device tables.
> 
> It does not appear so (none of the debug sections are present).  But I 
> could be wrong.
> 
> Stripping the modules on the penultimate Fedora 11 kernel results in the 
> same drop in size.  I can't imagine why a release kernel would have 
> anything extra left in the modules (unless it is just by accident).

Are you sure things still work after stripping?  Stuff like systemtap
and other tools?

thanks,

greg k-h

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 03:07 PM, Greg KH wrote:
> On Thu, Oct 01, 2009 at 02:56:55PM -0400, Nathaniel McCallum wrote:
>> On 10/01/2009 02:40 PM, Greg KH wrote:
>>> On Thu, Oct 01, 2009 at 02:35:40PM -0400, Nathaniel McCallum wrote:
>>>> On 10/01/2009 02:05 PM, Greg KH wrote:
>>>>> On Thu, Oct 01, 2009 at 01:01:50PM -0400, Nathaniel McCallum wrote:
>>>>>> On 10/01/2009 12:42 PM, Greg KH wrote:
>>>>>>> Why not just use the baseline kernel as a model for this.  Do a 'make
>>>>>>> allmodconfig' and then extract the data and publish it that way.  No
>>>>>>> kernel changes are needed, and then any distro can be easily matched up
>>>>>>> by this based on what they are using.  That will save you time in
>>>>>>> downloading zillions of distro releases, and provide a nice easy way to
>>>>>>> show what the kernel.org releases support.
>>>>>>
>>>>>> Unfortunately, I would not be able to track changes to the kernel in
>>>>>> this model.  Since this is one of my explicit goals (to make sure that
>>>>>> distro kernel changes get upstream), I think a non-invasive kernel
>>>>>> modification would be worth the effort.
>>>>>
>>>>> But this was an invasive modification, it added space to the kernel
>>>>> images for no real benifit other than for your tracking tools.  That's
>>>>> not going to fly unless you can find another good use for the change.
>>>>
>>>> Which is why I asked for advice for better options.  I would prefer a
>>>> non-invasive modification.  I am hoping that someone more familiar with
>>>> the layer would provide such a suggestion.
>>>>
>>>> One potential benefit for moving module info to ELF sections would be
>>>> the ability to strip kernel modules.  As a test, I did this on a recent
>>>> Fedora rawhide kernel I had lying around.  Stripping the modules results
>>>> in a 43% decrease in size (82M to 47M).
>>>
>>> Did those modules have debugging symbols enabled?  That seems like a
>>> large savings for just the module device tables.
>>
>> It does not appear so (none of the debug sections are present).  But I
>> could be wrong.
>>
>> Stripping the modules on the penultimate Fedora 11 kernel results in the
>> same drop in size.  I can't imagine why a release kernel would have
>> anything extra left in the modules (unless it is just by accident).
>
> Are you sure things still work after stripping?  Stuff like systemtap
> and other tools?

Not at all.  I have no idea what the ramifications are.

My point was merely that whatever is stripped out of the kernel should 
be able (in theory) to be stripped from the modules.  The only point of 
difference should be the actual module symbols themselves which can in 
stead be stored in ELF sections.

Nathaniel

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 02:40 PM, Greg KH wrote:
> On Thu, Oct 01, 2009 at 02:35:40PM -0400, Nathaniel McCallum wrote:
>> But I still would prefer a non-invasive solution.
>
> Agreed, sorry I don't have an idea on how to do it.

How about this?

-#define MODULE_GENERIC_TABLE(gtype,name)
+#define MODULE_GENERIC_TABLE(gtype,name)               \
+static const struct gtype##_id gtype##_id_table        \
+  __attribute__ ((unused, alias(__stringify(name))))

We add an alias to the existing symbol.  This just creates a new entry 
in the symbol table.  These entries are all stripped anyway and inserted 
into the System.map.

Impact:
	* No change to the resulting vmlinuz binary
	* Increases size of System.map (~2k with the Fedora config)

Nathaniel

Re: Exposing device ids and driver names

[Stefan Richter]

Nathaniel McCallum wrote:
> Please CC me on any responses as I'm not subscribed to lkml.
> 
> I have the aim at creating two tools helpful to linux.  The first tool 
> is a driver regression test of sorts.  I want to be able to create 
> essentially a time line of hardware support as they appear in distros. 
> The second tool, related to the first, is a program which runs on 
> Windows and scans for a user's hardware and tells them which distro will 
> best support their hardware.  I already have a working prototype of 
> these two tools.  It currently uses the data exported by modinfo.  This 
> however does not provide transparency for drivers compiled into the kernel.

Hardware support also depends on userland:  Udev rules, libraries,
application programs.

Even if you ignore that for now and only look at the kernel part of
hardware support:  Beyond "doesn't have a matching driver" and "does
have", there is a large and impossible to track grey area of "has a
poorly working driver" and "has a perfectly working driver".
-- 
Stefan Richter
-=====-==--= =-=- ----=
http://arcgraph.de/sr/

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 01:47 PM, Stefan Richter wrote:
> Nathaniel McCallum wrote:
>> Please CC me on any responses as I'm not subscribed to lkml.
>>
>> I have the aim at creating two tools helpful to linux.  The first tool
>> is a driver regression test of sorts.  I want to be able to create
>> essentially a time line of hardware support as they appear in distros.
>> The second tool, related to the first, is a program which runs on
>> Windows and scans for a user's hardware and tells them which distro will
>> best support their hardware.  I already have a working prototype of
>> these two tools.  It currently uses the data exported by modinfo.  This
>> however does not provide transparency for drivers compiled into the kernel.
>
> Hardware support also depends on userland:  Udev rules, libraries,
> application programs.
>
> Even if you ignore that for now and only look at the kernel part of
> hardware support:  Beyond "doesn't have a matching driver" and "does
> have", there is a large and impossible to track grey area of "has a
> poorly working driver" and "has a perfectly working driver".

Yes, I'm aware of this and will account for it as best as I am able. 
There is still tremendous value in identifying (with real data):
1. differences between distros and upstream
2. the flow of driver support between distros and upstream
3. statistically common devices that have no driver

Further, for the most common device classes (video cards, networking, 
wireless, etc) it is fairly easy to get a good reading.  For instance, 
the difference between "no graphics at all" and "some graphics" is 
infinitely larger than "some graphics" and "great 3D acceleration."  I'm 
not aware of any video cards which have drivers that claim to support 
them that can't at least modeset.

To summarize, you are correct that we cannot predict 100% of user 
experience.  But 70% is a huge improvement over 0%.

Nathaniel

Re: Exposing device ids and driver names

[Stefan Richter]

Nathaniel McCallum wrote:
> On 10/01/2009 01:47 PM, Stefan Richter wrote:
>> Nathaniel McCallum wrote:
[...]
>>> The second tool, related to the first, is a program which runs on
>>> Windows and scans for a user's hardware and tells them which distro will
>>> best support their hardware.
[...]
>> Hardware support also depends on userland:  Udev rules, libraries,
>> application programs.
>>
>> Even if you ignore that for now and only look at the kernel part of
>> hardware support:  Beyond "doesn't have a matching driver" and "does
>> have", there is a large and impossible to track grey area of "has a
>> poorly working driver" and "has a perfectly working driver".

There are even more factors for how well something works or even whether
it works at all:  It may even depend on combinations of two pieces of
hardware, e.g. bus adapter and device on that bus.  It may depend on
device firmware revisions.

> Yes, I'm aware of this and will account for it as best as I am able.
[...]
> you are correct that we cannot predict 100% of user
> experience.  But 70% is a huge improvement over 0%.

So, this 2nd tool can't literally say which distribution supports a
device best.  It can mostly just list which distributions contain a
matching kernel driver.
-- 
Stefan Richter
-=====-==--= =-=- ----=
http://arcgraph.de/sr/

Re: Exposing device ids and driver names

[Nathaniel McCallum]

On 10/01/2009 02:23 PM, Stefan Richter wrote:
> Nathaniel McCallum wrote:
>> On 10/01/2009 01:47 PM, Stefan Richter wrote:
>>> Nathaniel McCallum wrote:
> [...]
>>>> The second tool, related to the first, is a program which runs on
>>>> Windows and scans for a user's hardware and tells them which distro will
>>>> best support their hardware.
> [...]
>>> Hardware support also depends on userland:  Udev rules, libraries,
>>> application programs.
>>>
>>> Even if you ignore that for now and only look at the kernel part of
>>> hardware support:  Beyond "doesn't have a matching driver" and "does
>>> have", there is a large and impossible to track grey area of "has a
>>> poorly working driver" and "has a perfectly working driver".
>
> There are even more factors for how well something works or even whether
> it works at all:  It may even depend on combinations of two pieces of
> hardware, e.g. bus adapter and device on that bus.  It may depend on
> device firmware revisions.
>
>> Yes, I'm aware of this and will account for it as best as I am able.
> [...]
>> you are correct that we cannot predict 100% of user
>> experience.  But 70% is a huge improvement over 0%.
>
> So, this 2nd tool can't literally say which distribution supports a
> device best.  It can mostly just list which distributions contain a
> matching kernel driver.

I never said that.  What I said is that I will "do my best." This will 
include using other data sources, such as Xorg drivers, libusb drivers, 
user feedback, etc.

Nathaniel