Device Registry (DevReg) Patch 0.2.0

Device Registry (DevReg) Patch 0.2.0

[Tim Jansen]

The Linux Device Registry (devreg) is a kernel patch that adds a device 
database in XML format to the /proc filesystem. It collects all information 
about the system's physical devices, creates persistent device ids and 
provides them in the file /proc/devreg.

Devreg has three purposes:
- collect all configuration data from drivers so the user can browse his 
hardware configuration. 
-allow an application to display all devices that provide a certain interface 
(for example all mice) so the user can chose one. 
-allow an application to find the device that the user has selected after a 
reboot or a hotplug action: the device files in /dev do not offer stable 
names, they depend on the order in that the devices have been plugged in or 
powered on. 

Changes since last release (0.1.1):
- converted file format to XML
- bus-specific information from pci and usb added
- fixed locking

The patch (for 2.4.3) can be found at 
http://www.tjansen.de/devreg/devreg-2.4.3-0.2.0.diff.gz
To test it, apply the patch, select CONFIG_DEVFS_FS and CONFIG_DEVREG and 
compile. Note that the patch will break binary drivers.

Supported hardware in version 0.2.0: PCI subsystem, USB subsystem, most PCI 
sound cards, USB HID devices, USB hubs, USB printers

Other information and a user-space library can be found at 
http://www.tjansen.de/devreg

Re: Device Registry (DevReg) Patch 0.2.0

[Martin Dalecki]

Tim Jansen wrote:
> 
> The Linux Device Registry (devreg) is a kernel patch that adds a device
> database in XML format to the /proc filesystem. It collects all information
OH SHIT!!     ^^^ 

<IRONY>
Why don't you just add postscript output to /proc?
</IRONY>

> about the system's physical devices, creates persistent device ids and
> provides them in the file /proc/devreg.

Re: Device Registry (DevReg) Patch 0.2.0

[Tim Jansen]

On Tuesday 24 April 2001 11:40, Martin Dalecki wrote:
> Tim Jansen wrote:
> > The Linux Device Registry (devreg) is a kernel patch that adds a device
> > database in XML format to the /proc filesystem. It collects all
> OH SHIT!!      ^^^
> Why don't you just add postscript output to /proc?

XML wasn't my first choice. The 0.1.x versions used simple name/value pairs, 
I gave this up after trying to fit the complex USB 
configuration/interface/endpoint data into name/value pairs. Thinking about 
text file formats that allow me to display hierarchical information,  XML was 
the obvious choice for me. Are there alternatives to get complex and 
extendable information out to user space? (see 
http://www.tjansen.de/devreg/devreg.output.txt for a example /proc/devreg 
output)
My other ideas were:
- using a simple binary format, just dump structs. This would break all 
applications every time somebody changes the format, and this should happen 
very often because of the nature of the format
- using a complicated, extendable binary format, for example chunk-based like 
(a|r)iff file formats. This would add more code in the kernel than XML 
output, is difficult to understand and requires more work in user space 
(because XML parsers are already available)
- making up a new text-based format with properties similar to XML because I 
knew that many people dont like the idea of XML output in the kernel.. I 
really thought about it, but it does not make much sense.

The actual code overhead of XML output compared to a format like 
/proc/bus/usb/devices is almost zero, XML is only a little bit more verbose. 
I agree that XML is not perfect for this kind of data, but it is simple to 
generate, well known and I dont see a better alternative. 

bye..
 

Re: Device Registry (DevReg) Patch 0.2.0

[Martin Dalecki]

Tim Jansen wrote:
> 
> On Tuesday 24 April 2001 11:40, Martin Dalecki wrote:
> > Tim Jansen wrote:
> > > The Linux Device Registry (devreg) is a kernel patch that adds a device
> > > database in XML format to the /proc filesystem. It collects all
> > OH SHIT!!      ^^^
> > Why don't you just add postscript output to /proc?
> 
> XML wasn't my first choice. The 0.1.x versions used simple name/value pairs,
> I gave this up after trying to fit the complex USB
> configuration/interface/endpoint data into name/value pairs. Thinking about
> text file formats that allow me to display hierarchical information,  XML was
> the obvious choice for me. Are there alternatives to get complex and
> extendable information out to user space? (see
> http://www.tjansen.de/devreg/devreg.output.txt for a example /proc/devreg
> output)

Yes filesystem structures. Or just simple parsing in the user space
plain binary
data.

> My other ideas were:
> - using a simple binary format, just dump structs. This would break all
> applications every time somebody changes the format, and this should happen
> very often because of the nature of the format
> - using a complicated, extendable binary format, for example chunk-based like
> (a|r)iff file formats. This would add more code in the kernel than XML
> output, is difficult to understand and requires more work in user space
> (because XML parsers are already available)
> - making up a new text-based format with properties similar to XML because I
> knew that many people dont like the idea of XML output in the kernel.. I
> really thought about it, but it does not make much sense.
> 
> The actual code overhead of XML output compared to a format like
> /proc/bus/usb/devices is almost zero, XML is only a little bit more verbose.
> I agree that XML is not perfect for this kind of data, but it is simple to
> generate, well known and I dont see a better alternative.
> 
> bye..
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
- phone: +49 214 8656 283
- job:   eVision-Ventures AG, LEV .de (MY OPINIONS ARE MY OWN!)
- langs: de_DE.ISO8859-1, en_US, pl_PL.ISO8859-2, last ressort:
ru_RU.KOI8-R

Re: Device Registry (DevReg) Patch 0.2.0

[mirabilos]

> > > The Linux Device Registry (devreg) is a kernel patch that adds a
device
> > > database in XML format to the /proc filesystem. It collects all
> > OH SHIT!!      ^^^
> > Why don't you just add postscript output to /proc?
>
> XML wasn't my first choice. The 0.1.x versions used simple name/value
pairs,
> I gave this up after trying to fit the complex USB
> configuration/interface/endpoint data into name/value pairs. Thinking
about
> text file formats that allow me to display hierarchical information,
XML was
> the obvious choice for me. Are there alternatives to get complex and
> extendable information out to user space? (see
> http://www.tjansen.de/devreg/devreg.output.txt for a example
/proc/devreg
> output)
> My other ideas were:
> - using a simple binary format, just dump structs. This would break
all
> applications every time somebody changes the format, and this should
happen
> very often because of the nature of the format
> - using a complicated, extendable binary format, for example
chunk-based like
> (a|r)iff file formats. This would add more code in the kernel than XML
> output, is difficult to understand and requires more work in user
space
> (because XML parsers are already available)
> - making up a new text-based format with properties similar to XML
because I
> knew that many people dont like the idea of XML output in the kernel..
I
> really thought about it, but it does not make much sense.

What about indenting? I think of 0 spaces before the device name,
1 space before properties which belong to the device. (Is one level
enough? I'm currently offline so didn't check the sample)
Structure per entry:
   [Space] Name colon property
It also could be an equality sign, but then we could use no
indention at all and [] for the sections, which leaves us
at .INI format, which after all still is lotta more readable after
cat than XML.

> The actual code overhead of XML output compared to a format like
> /proc/bus/usb/devices is almost zero, XML is only a little bit more
verbose.
> I agree that XML is not perfect for this kind of data, but it is
simple to
> generate, well known and I dont see a better alternative.
>
> bye..

-mirabilos

Re: Device Registry (DevReg) Patch 0.2.0

[Tim Jansen]

On Tuesday 24 April 2001 18:39, Martin Dalecki wrote:
> Are there alternatives to get complex and extendable information out to 
> user space?
> Yes filesystem structures. 


How exactly can this work? A single value per file is not very helpful if you 
have a thousand values. You could cluster them (for example one level in the 
XML hierarchy == one file), but this will soon get very complicated. Its much 
more work to implement in the kernel, its painful in user-space and you cant 
just use a text editor to look at it (because you always have to look at 10 
files per device). 
IMHO only a single XML file per physical device is an option, but I do not 
know how to name the files...


> Or just simple parsing in the user space plain binary data.

This would be a compatibility nightmare and hard to maintain. Once you 
decided for a binary format you cannot change or extend it without breaking 
user-space apps. This may save a few lines code, but not many. All you need 
to add a line to XML output is a sprintf and a call to devreg_write_line(). 

One of the ideas of devreg is that while it has a common format for generic 
information, like the name and topology of physical devices, every driver can 
add additional data (this is why XML namespaces are used). Currently only the 
USB and PCI subsystems add data to devreg, but in future versions the device 
driver itself or other subsystems should do this, too. 

bye...

Re: Device Registry (DevReg) Patch 0.2.0

[Tim Jansen]

On Tuesday 24 April 2001 18:43, mirabilos wrote:
> What about indenting? I think of 0 spaces before the device name,
> 1 space before properties which belong to the device. 
> Structure per entry:
>    [Space] Name colon property

But what is the advantage? Its not less work in the kernel, and in user-space 
you need to write a parser for this. You would have made a new format for 
hierarchical data that no one else uses only to avoid using XML in the 
kernel. 


> Is one level enough? I'm currently offline so didn't check the sample

No, for example for USB you have the levels 
devices/configurations/interfaces/endpoints. 

bye...

Re: Device Registry (DevReg) Patch 0.2.0

[Dan Kegel]

Tim Jansen wrote:
> On Tuesday 24 April 2001 18:39, Martin Dalecki wrote: 
> >> Are there alternatives to get complex and extendable information out to 
> >> user space? 
> > Yes filesystem structures. 
> 
> How exactly can this work? A single value per file is not very helpful if you 
> have a thousand values. You could cluster them (for example one level in the 
> XML hierarchy == one file), but this will soon get very complicated. Its much 
> more work to implement in the kernel, its painful in user-space and you cant 
> just use a text editor to look at it (because you always have to look at 10 
> files per device). 

The command
  more foo/* foo/*/* 
will display the values in the foo subtree nicely, I think.

Think of the /proc tree as the XML parse tree already exploded for you.

The only problem with /proc as it stands is that there is no formal
syntax for its entries.  Some of them are hard to parse.

Before we add a new /proc entry that generates XML which summarizes
the rest of /proc, it might make sense to standardize /proc entries
and write a regression test to verify they are formatted correctly.
It would then be trivial to write a /proc to XML converter which
ran solely in userspace.

See 
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.0/0506.html
and
http://marc.theaimsgroup.com/?l=linux-kernel&s=%2Fproc+xml

for prior discussion on the matter.

I don't want to dismiss the reasons you want to use XML for this,
but tread carefully, lest you duplicate lots of code and introduce
cruft.  Better to factor the XML part out to a userspace library...

- Dan

Re: Device Registry (DevReg) Patch 0.2.0

[H. Peter Anvin]

Followup to:  <3AE704FA.DCF1BEC6@kegel.com>
By author:    Dan Kegel <dank@kegel.com>
In newsgroup: linux.dev.kernel
> 
> The only problem with /proc as it stands is that there is no formal
> syntax for its entries.  Some of them are hard to parse.
> 

/proc/sys is probably the method to follow.  Every item is a datum of
a simple datatype.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

/proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 19:10, you wrote:
> The command
>   more foo/* foo/*/*
> will display the values in the foo subtree nicely, I think.

Unfortunately it displays only the values. Dumping numbers and strings 
without knowing their meaning (and probably not even the order) is not very 
useful.

> Better to factor the XML part out to a userspace library...

But the one-value per file approach is MORE work. It would be less work to 
create XML and factor out the directory structure in user-space :)
Devreg collects its data from the drivers, each driver should contribute the 
information that it can provide about the device.
Printing a few values in XML format using the functions from xmlprocfs is as 
easy as writing
proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
                get_portnum(usbdev), usbdev->maxchild);

Extending the devreg output with driver-specific data means registering a 
callback function that prints the driver's data. The driver should use its 
own XML namespace, so whatever the driver adds will not break any 
(well-written) user-space applications. The data is created on-demand, so the 
values can be dynamic and do not waste any space when devreg is not used. 

The code is easy to read and not larger than a solution that creates static 
/proc entries, and holding the data completely static would take much more 
memory. And it takes less code than a solution that would create the values 
in /proc dynamically because this would mean one callback per file or a 
complicated way to specify several values with a single callback. 

bye...

 

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Tim Jansen wrote:
> 
> On Wednesday 25 April 2001 19:10, you wrote:
> > The command
> >   more foo/* foo/*/*
> > will display the values in the foo subtree nicely, I think.
> 
> Unfortunately it displays only the values. Dumping numbers and strings
> without knowing their meaning (and probably not even the order) is not very
> useful.

The meanings should be implied by the filenames, which are displayed (try it).
The order is alphabetical by filename.

> But the one-value per file approach is MORE work. It would be less work to
> create XML and factor out the directory structure in user-space :)
> Devreg collects its data from the drivers, each driver should contribute the
> information that it can provide about the device.
> Printing a few values in XML format using the functions from xmlprocfs is as
> easy as writing
> proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
>                 get_portnum(usbdev), usbdev->maxchild);

The corresponding one-value-per-file approach can probably be made to
be a single call per value.  IMHO that's more useful; it means that
(once we agree on definitions) programs don't need to parse XML to
access this data; they can go straight to the node in the document object
model tree ( = /proc ).  Think of /proc as a preparsed XML tree
that hasn't been standardized yet.
 
> The code is easy to read and not larger than a solution that creates static
> /proc entries, and holding the data completely static would take much more
> memory. And it takes less code than a solution that would create the values
> in /proc dynamically because this would mean one callback per file or a
> complicated way to specify several values with a single callback.

... but XML parsing is something we don't want to force on people
when we can provide the same data in a pre-parsed, much easier to access
form, IMHO.

Have you bothered to go back and read the old discussions on this topic?

> The driver should use its 
> own XML namespace, so whatever the driver adds will not break any 
> (well-written) user-space applications.

Are you trying to avoid writing a DTD?  IMHO it would be better to
have a single DTD for the entire tree, rather than a separate 
anything-goes namespace for each driver.  Yes, this is more work,
but all the Linux drivers are tightly integrated into the kernel
source tree, we may as well have a tightly-integrated DTD documenting
what each block, serial, synch, etc. driver must provide.

I think we both agree that there needs to be an easy, standardized way
to access this data.  IMHO there's a lot of standardizing that needs
to happen before you can start writing code -- otherwise your new code
won't help, and we'll be in the same mess we're in now.

The DTD can apply to both the existing /proc form and any proposed XML form
of config info exported by the kernel; there should be an easy transformation
between them.  And it has to come first!

- Dan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> On Wednesday 25 April 2001 19:10, you wrote:
> > The command
> >   more foo/* foo/*/*
> > will display the values in the foo subtree nicely, I think.
> 
> Unfortunately it displays only the values. Dumping numbers and strings 
> without knowing their meaning (and probably not even the order) is not very 
> useful.
> 
> > Better to factor the XML part out to a userspace library...
> 
> But the one-value per file approach is MORE work. It would be less work to 
> create XML and factor out the directory structure in user-space :)
> Devreg collects its data from the drivers, each driver should contribute the 
> information that it can provide about the device.
> Printing a few values in XML format using the functions from xmlprocfs is as 
> easy as writing
> proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
>                 get_portnum(usbdev), usbdev->maxchild);
> 
> Extending the devreg output with driver-specific data means registering a 
> callback function that prints the driver's data. The driver should use its 
> own XML namespace, so whatever the driver adds will not break any 
> (well-written) user-space applications. The data is created on-demand, so the 
> values can be dynamic and do not waste any space when devreg is not used. 
> 
> The code is easy to read and not larger than a solution that creates static 
> /proc entries, and holding the data completely static would take much more 
> memory. And it takes less code than a solution that would create the values 
> in /proc dynamically because this would mean one callback per file or a 
> complicated way to specify several values with a single callback. 

Personally, I think

	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);

(or the string "dddd ddd" with d representing a digit)

is shorter (and faster) to parse with

	fscanf(input,"%d %d",&usbdev,&maxchild);

Than it would be to try parsing

	<usb:topology port="ddddd" portnum="dddd">

with an XML parser.

Sorry - XML is good for some things. It is not designed to be a
interface language between a kernel and user space.

I am NOT in favor of "one file per value", but structured data needs
to be written in a reasonable, concise manner. XML is intended for
communication between disparate systems in an exreemly precise manner
to allow some self documentation to be included when the communication
fails.

Even Lisp S expressions are easier :-)

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Jesse Pollard wrote:
> Personally, I think
>         proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> (or the string "dddd ddd" with d representing a digit)
> 
> is shorter (and faster) to parse with
>         fscanf(input,"%d %d",&usbdev,&maxchild);
> 
> Than it would be to try parsing
>         <usb:topology port="ddddd" portnum="dddd">
> with an XML parser.
> 
> Sorry - XML is good for some things. It is not designed to be a
> interface language between a kernel and user space.
> 
> I am NOT in favor of "one file per value", but structured data needs
> to be written in a reasonable, concise manner. XML is intended for
> communication between disparate systems in an exreemly precise manner
> to allow some self documentation to be included when the communication
> fails.

Agreed.  

But one thing XML provides (potentially) is a DTD that defines meanings and formats.  
IMHO the kernel needs something like this for /proc (though not in DTD format!).

Has anyone ever tried to write a formal syntax for all the entries
in /proc?   We have bits and pieces of /proc documentation in 
/usr/src/linux/Documentation, but nothing you could feed directly 
into a parser generator.  It'd be neat to have a good definition for /proc
in the LSB, and have an LSB conformance test that could look in
/proc and say "Yup, all the entries there conform to the spec and can
be parsed properly."

(http://www.pathname.com/fhs/2.2-beta/fhs-2.2-beta.txt mentions /proc,
but doesn't standardize any of it, except to suggest that /etc/mtab
can be a symbolic link to /proc/mounts.)
- Dan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 21:37, you wrote:
> Personally, I think
>> 	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> is shorter (and faster) to parse with
> 	fscanf(input,"%d %d",&usbdev,&maxchild);

Right, but what happens if you need to extend the format? For example 
somebody adds support for USB 2.0 to the kernel and you need to some new 
values. Then you would have the choice between changing the format and 
breaking applications or keeping the format and dont provide the additional 
information. 
With XML (or single-value-per-file) it is easy to tell application to ignore 
unknown tags (or files). When you just list values you will be damned sooner 
or later, unless you make up additional rules that say how apps should handle 
these cases. And then your approach is no longer simple, but possibly even 
more complicated

bye... 

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Jesse Pollard]

Tim Jansen <tim@tjansen.de>:
> On Wednesday 25 April 2001 21:37, you wrote:
> > Personally, I think
> >> 	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> > is shorter (and faster) to parse with
> > 	fscanf(input,"%d %d",&usbdev,&maxchild);
> 
> Right, but what happens if you need to extend the format? For example 
> somebody adds support for USB 2.0 to the kernel and you need to some new 
> values. Then you would have the choice between changing the format and 
> breaking applications or keeping the format and dont provide the additional 
> information. 
> With XML (or single-value-per-file) it is easy to tell application to ignore 
> unknown tags (or files). When you just list values you will be damned sooner 
> or later, unless you make up additional rules that say how apps should handle 
> these cases. And then your approach is no longer simple, but possibly even 
> more complicated

Not necessarily. If the "extended data" is put following the current data
(since the data is currently record oriented) just making the output
format longer will not/should not casue problems in reading the data.
Just look at FORTRAN for an example of a extensible input :-) More data
on the record will/should just be ignored. The only coding change might
be to use a fgets to read a record, followed by a sscanf to get the known
values.

Alternatively, you can always put one value per record:
	tag:value
	tag2:value2...

This is still simpler than XML to read, and to generate.

The problem with this and XML is the same - If the tag is no longer relevent
(or changes its name), then the output must either continue to include it, or
break applications that depend on that tag.

In all cases, atomic extraction of the structured data will be problematical
since there may be buffering issues in output. XML is very verbose, and the
tagged format better; but a series of values goes even farther...

Try them out - Just go through the /proc/net formats and stick in the
XML... Just don't count on the regular utilities to decode them. It would
give some actual results to compair with the current structure.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[J . A . Magallon]

On 04.25 Jesse Pollard wrote:
> 
> Alternatively, you can always put one value per record:
> 	tag:value
> 	tag2:value2...
> 
> This is still simpler than XML to read, and to generate.
> 

Just my two cents.

It looks clear that /proc is for programs, not for humans. So the best format
for proc is just binary values. So programs can read it quickly, even in
a chunk if they know the format. But sometimes it is usefull to do a cat on
a /proc entry.

Question: it is possible to redirect the same fs call (say read) to different
implementations, based on the open mode of the file descriptor ? So, if
you open the entry in binary, you just get the number chunk, if you open
it in ascii you get a pretty printed version, or a format description like
Bus: %d
Device: %h
..
to 'vprintf' the values.

-- 
J.A. Magallon                                          #  Let the source
mailto:jamagallon@able.es                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Doug McNaught]

"J . A . Magallon" <jamagallon@able.es> writes:

> Question: it is possible to redirect the same fs call (say read) to different
> implementations, based on the open mode of the file descriptor ? So, if
> you open the entry in binary, you just get the number chunk, if you open
> it in ascii you get a pretty printed version, or a format description like

There is no distinction between "text" and "binary" modes on a file
descriptor.  The distinction exists in the C stdio layer, but is a
no-op on Unix systems.

-Doug
-- 
The rain man gave me two cures; he said jump right in,
The first was Texas medicine--the second was just railroad gin,
And like a fool I mixed them, and it strangled up my mind,
Now people just get uglier, and I got no sense of time...          --Dylan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[J . A . Magallon]

On 04.25 Doug McNaught wrote:
> "J . A . Magallon" <jamagallon@able.es> writes:
> 
> > Question: it is possible to redirect the same fs call (say read) to
> different
> > implementations, based on the open mode of the file descriptor ? So, if
> > you open the entry in binary, you just get the number chunk, if you open
> > it in ascii you get a pretty printed version, or a format description like
> 
> There is no distinction between "text" and "binary" modes on a file
> descriptor.  The distinction exists in the C stdio layer, but is a
> no-op on Unix systems.
> 

Yep, realized after the post, fopen() is a wrapper for open(). The idea
is to (someway) set the proc entry in verbose vs fast-binary mode for
reads. Perhaps an ioctl() or an fcntl() or something similar.
So the verbose mode gives the field names, and the binary mode just
gives the numbers. Applications that know what are reading can just
read binary data, and fast.

-- 
J.A. Magallon                                          #  Let the source
mailto:jamagallon@able.es                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Marko Kreen]

On Thu, Apr 26, 2001 at 12:03:25AM +0200, J . A . Magallon wrote:
> 
> On 04.25 Doug McNaught wrote:
> > "J . A . Magallon" <jamagallon@able.es> writes:
> > 
> > > Question: it is possible to redirect the same fs call (say read) to
> > different
> > > implementations, based on the open mode of the file descriptor ? So, if
> > > you open the entry in binary, you just get the number chunk, if you open
> > > it in ascii you get a pretty printed version, or a format description like
> > 
> > There is no distinction between "text" and "binary" modes on a file
> > descriptor.  The distinction exists in the C stdio layer, but is a
> > no-op on Unix systems.
> > 
> 
> Yep, realized after the post, fopen() is a wrapper for open(). The idea
> is to (someway) set the proc entry in verbose vs fast-binary mode for
> reads. Perhaps an ioctl() or an fcntl() or something similar.
> So the verbose mode gives the field names, and the binary mode just
> gives the numbers. Applications that know what are reading can just
> read binary data, and fast.

Eh.  Search in archives for "ascii is tough"...

-- 
marko

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Mark Hahn]

> > Question: it is possible to redirect the same fs call (say read) to different
> > implementations, based on the open mode of the file descriptor ? So, if
> > you open the entry in binary, you just get the number chunk, if you open
> > it in ascii you get a pretty printed version, or a format description like
> 
> There is no distinction between "text" and "binary" modes on a file
> descriptor.  The distinction exists in the C stdio layer, but is a
> no-op on Unix systems.

of course.  but we could trivially define O_PROC_BINARY,
or an ioctl/fcntl, or even do something fancy like use lseek().

pardon my stream of consciousness here, but:

I think it's well-established that proc exists for humans,
and that there's no real sympathy for the eternal whines of 
how terribly hard it is to parse.  it's NOT hard to parse,
but would be more trivial if it were more consistent.

the main goal at this point is to make kernel proc-related 
code more efficient, easy-to-use, etc.  a purely secondary goal
is to make user-space tools more robust, efficient, and simpler.

there are three things that need to be communicated through the proc
interface, for each chunk of data: its type, it's name and its value.
it's critical that data be tagged in some way, since that's the only
way to permit back-compatibility.  that is, a tool looking for a particular
tag will naturally ignore new data with other tags.

/proc/sys is an attempt to provide tagged data; it works well, is 
easy to comprehend, but requires an open for each datum, and provides
no hints about type.

/proc/cpuinfo is another attempt: "tag : data", with no attempt to
provide types.  the tags have also mutated somewhat over time.

/proc/partitions is an example of a record-oriented file:
one line per record, and tags for the record members at the top.
still no typing information.

I have a sense that all of these could be collapsed into a single
api where kernel systems would register hierarchies of tuples of
<type,tag,callback>, where callback would be passed the tag,
and proc code would take care of "rendering" the data into 
human readable text (default), binary, or even xml.  the latter
would require some signalling mechanism like O_PROC_XML or the like.
further, programs could perform a meta-query, where they ask for
the types and tags of a datum (or hierarchy), so that on subsequent
queries, they'd now how to handle binary data.

if only one piece of code handled the rendering of /proc stuff,
it could do more, without burdoning all the disparate /proc producers.

regards, mark hahn.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Alexander Viro]

On Thu, 26 Apr 2001, J . A . Magallon wrote:

> 
> On 04.25 Doug McNaught wrote:
> > "J . A . Magallon" <jamagallon@able.es> writes:
> > 
> > > Question: it is possible to redirect the same fs call (say read) to
> > different
> > > implementations, based on the open mode of the file descriptor ? So, if
> > > you open the entry in binary, you just get the number chunk, if you open
> > > it in ascii you get a pretty printed version, or a format description like
> > 
> > There is no distinction between "text" and "binary" modes on a file
> > descriptor.  The distinction exists in the C stdio layer, but is a
> > no-op on Unix systems.
> > 
> 
> Yep, realized after the post, fopen() is a wrapper for open(). The idea
> is to (someway) set the proc entry in verbose vs fast-binary mode for
> reads. Perhaps an ioctl() or an fcntl() or something similar.
> So the verbose mode gives the field names, and the binary mode just
> gives the numbers. Applications that know what are reading can just
> read binary data, and fast.

OK, _what_ applications spend a considerable time (and considerable
percentage of the total execution time) parsing stuff in /proc?
ps(1)? top(1)? Fine. They touch how many files outside of /proc/<pid>/* ?
Exactly.

_Please_, drop this idiotic "parsing ASCII is slow" strawman. Or show some
valid examples.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 23:16, you wrote:
> Not necessarily. If the "extended data" is put following the current data
> (since the data is currently record oriented) just making the output
> format longer will not/should not casue problems in reading the data.
> Alternatively, you can always put one value per record:
> 	tag:value
> 	tag2:value2...

Both solutions only work for simple data, they dont help for more complex 
things like adding a variable-sized list of structures. Actually the first 
devreg version used something like your second proposal and I gave it up 
because it wasnt flexible enough to add USB configuration data. 

bye...

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 21:19, you wrote:
> The corresponding one-value-per-file approach can probably be made to
> be a single call per value.  

Yes, the real problem is writing a callback-based filesystem (unless you want 
to hold everything in memory). After thinking about it for the last two hours 
I already find the one-value-per-file approach not as hard to do as I did 
before, but it's still a lot of work.


> Have you bothered to go back and read the old discussions on this topic?

Yes. But in my case is different than, for example, the files in /proc/sys:
- the file names in /proc/sys are static. For devreg the filenames must be 
made dynamically (similar to the /proc process directories or usbdevfs)
- in /proc/sys there is just one piece for code responsible for every file or 
directory and no cooperation between different parts. If devreg creates, for 
example, a directory for a USB mouse it must be prepared to share this 
directory with the USB subsystem, the input subsystem and the USB hid driver. 
All four modules are responsible for their own files. 
- files and their content should be created on demand, so there must be some 
callback to tell the USB subsystem something like "the user just opened the 
directory of device X, please tell me which directories or files you want to 
add". 

It is certainly possible to convert devreg to the one-value-per-file approach 
and if this is all that it takes to get into some future (2.5) kernel I will 
do it. I just doubt that this is the easiest way to implement the 
functionality, because that's what I really want.


> Are you trying to avoid writing a DTD?  

Yes, at least a have a complete DTD, because it would be a nightmare to 
maintain it. Each time somebody adds a new capability to a driver the DTD 
would have to be updated. And what about drivers that are not part of the 
official kernel?
I thought about using a separate XML Schema definition for each namespace 
though.

bye...

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Thursday 26 April 2001 00:24, Mark Hahn wrote:
> I have a sense that all of these could be collapsed into a single
> api where kernel systems would register hierarchies of tuples of
> <type,tag,callback>, where callback would be passed the tag,

You also need to know the parent of the tuple to build a hierarchy. And it 
should be possible to create lists. 

The callback prototypes for values could look like this:

int proc_value_cb_string(char *buf, void *context); // writes string to buf, 
// returns len of string or negative value for error
int proc_value_cb_int(int *value, void *context); 


For parent/directory tuples you would provide two additional callbacks that 
set the context for their children and maybe take care of other things like 
locking (so they dont need to be done in every single value callback):

void *proc_value_cb_level_enter(void *old_context); // returns new context
void proc_value_cb_level_leave(void *old_context, void *new_context);


For tuples with a list there would be two callbacks to get the list elements:

int proc_value_cb_list_num(void *context); // returns number of elements
void *proc_value_cb_list_context(int index, void *context); // returns context
// of the element at the given index or NULL


To register such a tuple you would have the following functions:
void proc_value_register_string(parent_handle_t parent, 
                                             const char *name,
		                     proc_value_cb_string cb);
void proc_value_register_int(parent_handle_t parent, 
                                         const char *name,
			     proc_value_cb_int cb);
parent_handle_t proc_value_register_parent(parent_handle_t parent, 
                                  const char *name,
 		          proc_value_cb_level_enter cb1,
		          proc_value_cb_level_leave cb2);
parent_handle_t proc_value_register_list(parent_handle_t h, 
	     	          proc_value_cb_list_num cbnum,
	                      proc_value_cb_list_context cbcon);

This is the simplest API that I can imagine for this. The only problem is 
that you need to write a callback for each value (file). Just printing XML 
still looks easier to me...


> and proc code would take care of "rendering" the data into
> human readable text (default), binary, or even xml.  the latter
> would require some signalling mechanism like O_PROC_XML or the like.

Then you can argue that once you have a single format implemented in the 
kernel you can convert it to whatever you like in user-space. And it seems 
like the decision for "one-value-per-file" in /proc has already been made 
(please correct me if not and we start all over again), so I will try to make 
a generic API like the one above for it.


> further, programs could perform a meta-query, where they ask for
> the types and tags of a datum (or hierarchy), so that on subsequent
> queries, they'd now how to handle binary data.

That would undermine the only advantage of binary data: it's easy (and 
fast) to dump or read a C struct. Not that I would really care for binary 
data...

bye...