Re: Device Registry (DevReg) Patch 0.2.0

Re: Device Registry (DevReg) Patch 0.2.0

[Dan Kegel]

Tim Jansen wrote:
> On Tuesday 24 April 2001 18:39, Martin Dalecki wrote: 
> >> Are there alternatives to get complex and extendable information out to 
> >> user space? 
> > Yes filesystem structures. 
> 
> How exactly can this work? A single value per file is not very helpful if you 
> have a thousand values. You could cluster them (for example one level in the 
> XML hierarchy == one file), but this will soon get very complicated. Its much 
> more work to implement in the kernel, its painful in user-space and you cant 
> just use a text editor to look at it (because you always have to look at 10 
> files per device). 

The command
  more foo/* foo/*/* 
will display the values in the foo subtree nicely, I think.

Think of the /proc tree as the XML parse tree already exploded for you.

The only problem with /proc as it stands is that there is no formal
syntax for its entries.  Some of them are hard to parse.

Before we add a new /proc entry that generates XML which summarizes
the rest of /proc, it might make sense to standardize /proc entries
and write a regression test to verify they are formatted correctly.
It would then be trivial to write a /proc to XML converter which
ran solely in userspace.

See 
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.0/0506.html
and
http://marc.theaimsgroup.com/?l=linux-kernel&s=%2Fproc+xml

for prior discussion on the matter.

I don't want to dismiss the reasons you want to use XML for this,
but tread carefully, lest you duplicate lots of code and introduce
cruft.  Better to factor the XML part out to a userspace library...

- Dan

Re: Device Registry (DevReg) Patch 0.2.0

[H. Peter Anvin]

Followup to:  <3AE704FA.DCF1BEC6@kegel.com>
By author:    Dan Kegel <dank@kegel.com>
In newsgroup: linux.dev.kernel
> 
> The only problem with /proc as it stands is that there is no formal
> syntax for its entries.  Some of them are hard to parse.
> 

/proc/sys is probably the method to follow.  Every item is a datum of
a simple datatype.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

/proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 19:10, you wrote:
> The command
>   more foo/* foo/*/*
> will display the values in the foo subtree nicely, I think.

Unfortunately it displays only the values. Dumping numbers and strings 
without knowing their meaning (and probably not even the order) is not very 
useful.

> Better to factor the XML part out to a userspace library...

But the one-value per file approach is MORE work. It would be less work to 
create XML and factor out the directory structure in user-space :)
Devreg collects its data from the drivers, each driver should contribute the 
information that it can provide about the device.
Printing a few values in XML format using the functions from xmlprocfs is as 
easy as writing
proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
                get_portnum(usbdev), usbdev->maxchild);

Extending the devreg output with driver-specific data means registering a 
callback function that prints the driver's data. The driver should use its 
own XML namespace, so whatever the driver adds will not break any 
(well-written) user-space applications. The data is created on-demand, so the 
values can be dynamic and do not waste any space when devreg is not used. 

The code is easy to read and not larger than a solution that creates static 
/proc entries, and holding the data completely static would take much more 
memory. And it takes less code than a solution that would create the values 
in /proc dynamically because this would mean one callback per file or a 
complicated way to specify several values with a single callback. 

bye...

 

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Tim Jansen wrote:
> 
> On Wednesday 25 April 2001 19:10, you wrote:
> > The command
> >   more foo/* foo/*/*
> > will display the values in the foo subtree nicely, I think.
> 
> Unfortunately it displays only the values. Dumping numbers and strings
> without knowing their meaning (and probably not even the order) is not very
> useful.

The meanings should be implied by the filenames, which are displayed (try it).
The order is alphabetical by filename.

> But the one-value per file approach is MORE work. It would be less work to
> create XML and factor out the directory structure in user-space :)
> Devreg collects its data from the drivers, each driver should contribute the
> information that it can provide about the device.
> Printing a few values in XML format using the functions from xmlprocfs is as
> easy as writing
> proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
>                 get_portnum(usbdev), usbdev->maxchild);

The corresponding one-value-per-file approach can probably be made to
be a single call per value.  IMHO that's more useful; it means that
(once we agree on definitions) programs don't need to parse XML to
access this data; they can go straight to the node in the document object
model tree ( = /proc ).  Think of /proc as a preparsed XML tree
that hasn't been standardized yet.
 
> The code is easy to read and not larger than a solution that creates static
> /proc entries, and holding the data completely static would take much more
> memory. And it takes less code than a solution that would create the values
> in /proc dynamically because this would mean one callback per file or a
> complicated way to specify several values with a single callback.

... but XML parsing is something we don't want to force on people
when we can provide the same data in a pre-parsed, much easier to access
form, IMHO.

Have you bothered to go back and read the old discussions on this topic?

> The driver should use its 
> own XML namespace, so whatever the driver adds will not break any 
> (well-written) user-space applications.

Are you trying to avoid writing a DTD?  IMHO it would be better to
have a single DTD for the entire tree, rather than a separate 
anything-goes namespace for each driver.  Yes, this is more work,
but all the Linux drivers are tightly integrated into the kernel
source tree, we may as well have a tightly-integrated DTD documenting
what each block, serial, synch, etc. driver must provide.

I think we both agree that there needs to be an easy, standardized way
to access this data.  IMHO there's a lot of standardizing that needs
to happen before you can start writing code -- otherwise your new code
won't help, and we'll be in the same mess we're in now.

The DTD can apply to both the existing /proc form and any proposed XML form
of config info exported by the kernel; there should be an easy transformation
between them.  And it has to come first!

- Dan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 21:19, you wrote:
> The corresponding one-value-per-file approach can probably be made to
> be a single call per value.  

Yes, the real problem is writing a callback-based filesystem (unless you want 
to hold everything in memory). After thinking about it for the last two hours 
I already find the one-value-per-file approach not as hard to do as I did 
before, but it's still a lot of work.


> Have you bothered to go back and read the old discussions on this topic?

Yes. But in my case is different than, for example, the files in /proc/sys:
- the file names in /proc/sys are static. For devreg the filenames must be 
made dynamically (similar to the /proc process directories or usbdevfs)
- in /proc/sys there is just one piece for code responsible for every file or 
directory and no cooperation between different parts. If devreg creates, for 
example, a directory for a USB mouse it must be prepared to share this 
directory with the USB subsystem, the input subsystem and the USB hid driver. 
All four modules are responsible for their own files. 
- files and their content should be created on demand, so there must be some 
callback to tell the USB subsystem something like "the user just opened the 
directory of device X, please tell me which directories or files you want to 
add". 

It is certainly possible to convert devreg to the one-value-per-file approach 
and if this is all that it takes to get into some future (2.5) kernel I will 
do it. I just doubt that this is the easiest way to implement the 
functionality, because that's what I really want.


> Are you trying to avoid writing a DTD?  

Yes, at least a have a complete DTD, because it would be a nightmare to 
maintain it. Each time somebody adds a new capability to a driver the DTD 
would have to be updated. And what about drivers that are not part of the 
official kernel?
I thought about using a separate XML Schema definition for each namespace 
though.

bye...

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> On Wednesday 25 April 2001 19:10, you wrote:
> > The command
> >   more foo/* foo/*/*
> > will display the values in the foo subtree nicely, I think.
> 
> Unfortunately it displays only the values. Dumping numbers and strings 
> without knowing their meaning (and probably not even the order) is not very 
> useful.
> 
> > Better to factor the XML part out to a userspace library...
> 
> But the one-value per file approach is MORE work. It would be less work to 
> create XML and factor out the directory structure in user-space :)
> Devreg collects its data from the drivers, each driver should contribute the 
> information that it can provide about the device.
> Printing a few values in XML format using the functions from xmlprocfs is as 
> easy as writing
> proc_printf(fragment, "<usb:topology port=\"%d\" portnum=\"%d\"/>\n",
>                 get_portnum(usbdev), usbdev->maxchild);
> 
> Extending the devreg output with driver-specific data means registering a 
> callback function that prints the driver's data. The driver should use its 
> own XML namespace, so whatever the driver adds will not break any 
> (well-written) user-space applications. The data is created on-demand, so the 
> values can be dynamic and do not waste any space when devreg is not used. 
> 
> The code is easy to read and not larger than a solution that creates static 
> /proc entries, and holding the data completely static would take much more 
> memory. And it takes less code than a solution that would create the values 
> in /proc dynamically because this would mean one callback per file or a 
> complicated way to specify several values with a single callback. 

Personally, I think

	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);

(or the string "dddd ddd" with d representing a digit)

is shorter (and faster) to parse with

	fscanf(input,"%d %d",&usbdev,&maxchild);

Than it would be to try parsing

	<usb:topology port="ddddd" portnum="dddd">

with an XML parser.

Sorry - XML is good for some things. It is not designed to be a
interface language between a kernel and user space.

I am NOT in favor of "one file per value", but structured data needs
to be written in a reasonable, concise manner. XML is intended for
communication between disparate systems in an exreemly precise manner
to allow some self documentation to be included when the communication
fails.

Even Lisp S expressions are easier :-)

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Jesse Pollard wrote:
> Personally, I think
>         proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> (or the string "dddd ddd" with d representing a digit)
> 
> is shorter (and faster) to parse with
>         fscanf(input,"%d %d",&usbdev,&maxchild);
> 
> Than it would be to try parsing
>         <usb:topology port="ddddd" portnum="dddd">
> with an XML parser.
> 
> Sorry - XML is good for some things. It is not designed to be a
> interface language between a kernel and user space.
> 
> I am NOT in favor of "one file per value", but structured data needs
> to be written in a reasonable, concise manner. XML is intended for
> communication between disparate systems in an exreemly precise manner
> to allow some self documentation to be included when the communication
> fails.

Agreed.  

But one thing XML provides (potentially) is a DTD that defines meanings and formats.  
IMHO the kernel needs something like this for /proc (though not in DTD format!).

Has anyone ever tried to write a formal syntax for all the entries
in /proc?   We have bits and pieces of /proc documentation in 
/usr/src/linux/Documentation, but nothing you could feed directly 
into a parser generator.  It'd be neat to have a good definition for /proc
in the LSB, and have an LSB conformance test that could look in
/proc and say "Yup, all the entries there conform to the spec and can
be parsed properly."

(http://www.pathname.com/fhs/2.2-beta/fhs-2.2-beta.txt mentions /proc,
but doesn't standardize any of it, except to suggest that /etc/mtab
can be a symbolic link to /proc/mounts.)
- Dan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 21:37, you wrote:
> Personally, I think
>> 	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> is shorter (and faster) to parse with
> 	fscanf(input,"%d %d",&usbdev,&maxchild);

Right, but what happens if you need to extend the format? For example 
somebody adds support for USB 2.0 to the kernel and you need to some new 
values. Then you would have the choice between changing the format and 
breaking applications or keeping the format and dont provide the additional 
information. 
With XML (or single-value-per-file) it is easy to tell application to ignore 
unknown tags (or files). When you just list values you will be damned sooner 
or later, unless you make up additional rules that say how apps should handle 
these cases. And then your approach is no longer simple, but possibly even 
more complicated

bye... 

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Jesse Pollard]

Tim Jansen <tim@tjansen.de>:
> On Wednesday 25 April 2001 21:37, you wrote:
> > Personally, I think
> >> 	proc_printf(fragment, "%d %d",get_portnum(usbdev), usbdev->maxchild);
> > is shorter (and faster) to parse with
> > 	fscanf(input,"%d %d",&usbdev,&maxchild);
> 
> Right, but what happens if you need to extend the format? For example 
> somebody adds support for USB 2.0 to the kernel and you need to some new 
> values. Then you would have the choice between changing the format and 
> breaking applications or keeping the format and dont provide the additional 
> information. 
> With XML (or single-value-per-file) it is easy to tell application to ignore 
> unknown tags (or files). When you just list values you will be damned sooner 
> or later, unless you make up additional rules that say how apps should handle 
> these cases. And then your approach is no longer simple, but possibly even 
> more complicated

Not necessarily. If the "extended data" is put following the current data
(since the data is currently record oriented) just making the output
format longer will not/should not casue problems in reading the data.
Just look at FORTRAN for an example of a extensible input :-) More data
on the record will/should just be ignored. The only coding change might
be to use a fgets to read a record, followed by a sscanf to get the known
values.

Alternatively, you can always put one value per record:
	tag:value
	tag2:value2...

This is still simpler than XML to read, and to generate.

The problem with this and XML is the same - If the tag is no longer relevent
(or changes its name), then the output must either continue to include it, or
break applications that depend on that tag.

In all cases, atomic extraction of the structured data will be problematical
since there may be buffering issues in output. XML is very verbose, and the
tagged format better; but a series of values goes even farther...

Try them out - Just go through the /proc/net formats and stick in the
XML... Just don't count on the regular utilities to decode them. It would
give some actual results to compair with the current structure.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[J . A . Magallon]

On 04.25 Jesse Pollard wrote:
> 
> Alternatively, you can always put one value per record:
> 	tag:value
> 	tag2:value2...
> 
> This is still simpler than XML to read, and to generate.
> 

Just my two cents.

It looks clear that /proc is for programs, not for humans. So the best format
for proc is just binary values. So programs can read it quickly, even in
a chunk if they know the format. But sometimes it is usefull to do a cat on
a /proc entry.

Question: it is possible to redirect the same fs call (say read) to different
implementations, based on the open mode of the file descriptor ? So, if
you open the entry in binary, you just get the number chunk, if you open
it in ascii you get a pretty printed version, or a format description like
Bus: %d
Device: %h
..
to 'vprintf' the values.

-- 
J.A. Magallon                                          #  Let the source
mailto:jamagallon@able.es                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Doug McNaught]

"J . A . Magallon" <jamagallon@able.es> writes:

> Question: it is possible to redirect the same fs call (say read) to different
> implementations, based on the open mode of the file descriptor ? So, if
> you open the entry in binary, you just get the number chunk, if you open
> it in ascii you get a pretty printed version, or a format description like

There is no distinction between "text" and "binary" modes on a file
descriptor.  The distinction exists in the C stdio layer, but is a
no-op on Unix systems.

-Doug
-- 
The rain man gave me two cures; he said jump right in,
The first was Texas medicine--the second was just railroad gin,
And like a fool I mixed them, and it strangled up my mind,
Now people just get uglier, and I got no sense of time...          --Dylan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[J . A . Magallon]

On 04.25 Doug McNaught wrote:
> "J . A . Magallon" <jamagallon@able.es> writes:
> 
> > Question: it is possible to redirect the same fs call (say read) to
> different
> > implementations, based on the open mode of the file descriptor ? So, if
> > you open the entry in binary, you just get the number chunk, if you open
> > it in ascii you get a pretty printed version, or a format description like
> 
> There is no distinction between "text" and "binary" modes on a file
> descriptor.  The distinction exists in the C stdio layer, but is a
> no-op on Unix systems.
> 

Yep, realized after the post, fopen() is a wrapper for open(). The idea
is to (someway) set the proc entry in verbose vs fast-binary mode for
reads. Perhaps an ioctl() or an fcntl() or something similar.
So the verbose mode gives the field names, and the binary mode just
gives the numbers. Applications that know what are reading can just
read binary data, and fast.

-- 
J.A. Magallon                                          #  Let the source
mailto:jamagallon@able.es                              #  be with you, Luke... 

Linux werewolf 2.4.3-ac14 #1 SMP Wed Apr 25 02:07:45 CEST 2001 i686

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Marko Kreen]

On Thu, Apr 26, 2001 at 12:03:25AM +0200, J . A . Magallon wrote:
> 
> On 04.25 Doug McNaught wrote:
> > "J . A . Magallon" <jamagallon@able.es> writes:
> > 
> > > Question: it is possible to redirect the same fs call (say read) to
> > different
> > > implementations, based on the open mode of the file descriptor ? So, if
> > > you open the entry in binary, you just get the number chunk, if you open
> > > it in ascii you get a pretty printed version, or a format description like
> > 
> > There is no distinction between "text" and "binary" modes on a file
> > descriptor.  The distinction exists in the C stdio layer, but is a
> > no-op on Unix systems.
> > 
> 
> Yep, realized after the post, fopen() is a wrapper for open(). The idea
> is to (someway) set the proc entry in verbose vs fast-binary mode for
> reads. Perhaps an ioctl() or an fcntl() or something similar.
> So the verbose mode gives the field names, and the binary mode just
> gives the numbers. Applications that know what are reading can just
> read binary data, and fast.

Eh.  Search in archives for "ascii is tough"...

-- 
marko

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Alexander Viro]

On Thu, 26 Apr 2001, J . A . Magallon wrote:

> 
> On 04.25 Doug McNaught wrote:
> > "J . A . Magallon" <jamagallon@able.es> writes:
> > 
> > > Question: it is possible to redirect the same fs call (say read) to
> > different
> > > implementations, based on the open mode of the file descriptor ? So, if
> > > you open the entry in binary, you just get the number chunk, if you open
> > > it in ascii you get a pretty printed version, or a format description like
> > 
> > There is no distinction between "text" and "binary" modes on a file
> > descriptor.  The distinction exists in the C stdio layer, but is a
> > no-op on Unix systems.
> > 
> 
> Yep, realized after the post, fopen() is a wrapper for open(). The idea
> is to (someway) set the proc entry in verbose vs fast-binary mode for
> reads. Perhaps an ioctl() or an fcntl() or something similar.
> So the verbose mode gives the field names, and the binary mode just
> gives the numbers. Applications that know what are reading can just
> read binary data, and fast.

OK, _what_ applications spend a considerable time (and considerable
percentage of the total execution time) parsing stuff in /proc?
ps(1)? top(1)? Fine. They touch how many files outside of /proc/<pid>/* ?
Exactly.

_Please_, drop this idiotic "parsing ASCII is slow" strawman. Or show some
valid examples.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Mark Hahn]

> > Question: it is possible to redirect the same fs call (say read) to different
> > implementations, based on the open mode of the file descriptor ? So, if
> > you open the entry in binary, you just get the number chunk, if you open
> > it in ascii you get a pretty printed version, or a format description like
> 
> There is no distinction between "text" and "binary" modes on a file
> descriptor.  The distinction exists in the C stdio layer, but is a
> no-op on Unix systems.

of course.  but we could trivially define O_PROC_BINARY,
or an ioctl/fcntl, or even do something fancy like use lseek().

pardon my stream of consciousness here, but:

I think it's well-established that proc exists for humans,
and that there's no real sympathy for the eternal whines of 
how terribly hard it is to parse.  it's NOT hard to parse,
but would be more trivial if it were more consistent.

the main goal at this point is to make kernel proc-related 
code more efficient, easy-to-use, etc.  a purely secondary goal
is to make user-space tools more robust, efficient, and simpler.

there are three things that need to be communicated through the proc
interface, for each chunk of data: its type, it's name and its value.
it's critical that data be tagged in some way, since that's the only
way to permit back-compatibility.  that is, a tool looking for a particular
tag will naturally ignore new data with other tags.

/proc/sys is an attempt to provide tagged data; it works well, is 
easy to comprehend, but requires an open for each datum, and provides
no hints about type.

/proc/cpuinfo is another attempt: "tag : data", with no attempt to
provide types.  the tags have also mutated somewhat over time.

/proc/partitions is an example of a record-oriented file:
one line per record, and tags for the record members at the top.
still no typing information.

I have a sense that all of these could be collapsed into a single
api where kernel systems would register hierarchies of tuples of
<type,tag,callback>, where callback would be passed the tag,
and proc code would take care of "rendering" the data into 
human readable text (default), binary, or even xml.  the latter
would require some signalling mechanism like O_PROC_XML or the like.
further, programs could perform a meta-query, where they ask for
the types and tags of a datum (or hierarchy), so that on subsequent
queries, they'd now how to handle binary data.

if only one piece of code handled the rendering of /proc stuff,
it could do more, without burdoning all the disparate /proc producers.

regards, mark hahn.

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Thursday 26 April 2001 00:24, Mark Hahn wrote:
> I have a sense that all of these could be collapsed into a single
> api where kernel systems would register hierarchies of tuples of
> <type,tag,callback>, where callback would be passed the tag,

You also need to know the parent of the tuple to build a hierarchy. And it 
should be possible to create lists. 

The callback prototypes for values could look like this:

int proc_value_cb_string(char *buf, void *context); // writes string to buf, 
// returns len of string or negative value for error
int proc_value_cb_int(int *value, void *context); 


For parent/directory tuples you would provide two additional callbacks that 
set the context for their children and maybe take care of other things like 
locking (so they dont need to be done in every single value callback):

void *proc_value_cb_level_enter(void *old_context); // returns new context
void proc_value_cb_level_leave(void *old_context, void *new_context);


For tuples with a list there would be two callbacks to get the list elements:

int proc_value_cb_list_num(void *context); // returns number of elements
void *proc_value_cb_list_context(int index, void *context); // returns context
// of the element at the given index or NULL


To register such a tuple you would have the following functions:
void proc_value_register_string(parent_handle_t parent, 
                                             const char *name,
		                     proc_value_cb_string cb);
void proc_value_register_int(parent_handle_t parent, 
                                         const char *name,
			     proc_value_cb_int cb);
parent_handle_t proc_value_register_parent(parent_handle_t parent, 
                                  const char *name,
 		          proc_value_cb_level_enter cb1,
		          proc_value_cb_level_leave cb2);
parent_handle_t proc_value_register_list(parent_handle_t h, 
	     	          proc_value_cb_list_num cbnum,
	                      proc_value_cb_list_context cbcon);

This is the simplest API that I can imagine for this. The only problem is 
that you need to write a callback for each value (file). Just printing XML 
still looks easier to me...


> and proc code would take care of "rendering" the data into
> human readable text (default), binary, or even xml.  the latter
> would require some signalling mechanism like O_PROC_XML or the like.

Then you can argue that once you have a single format implemented in the 
kernel you can convert it to whatever you like in user-space. And it seems 
like the decision for "one-value-per-file" in /proc has already been made 
(please correct me if not and we start all over again), so I will try to make 
a generic API like the one above for it.


> further, programs could perform a meta-query, where they ask for
> the types and tags of a datum (or hierarchy), so that on subsequent
> queries, they'd now how to handle binary data.

That would undermine the only advantage of binary data: it's easy (and 
fast) to dump or read a C struct. Not that I would really care for binary 
data...

bye...

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Tim Jansen]

On Wednesday 25 April 2001 23:16, you wrote:
> Not necessarily. If the "extended data" is put following the current data
> (since the data is currently record oriented) just making the output
> format longer will not/should not casue problems in reading the data.
> Alternatively, you can always put one value per record:
> 	tag:value
> 	tag2:value2...

Both solutions only work for simple data, they dont help for more complex 
things like adding a variable-sized list of structures. Actually the first 
devreg version used something like your second proposal and I gave it up 
because it wasnt flexible enough to add USB configuration data. 

bye...

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Jesse Pollard wrote:
> > But one thing XML provides (potentially) is a DTD that defines meanings and formats.
> > IMHO the kernel needs something like this for /proc (though not in DTD format!).
> >
> > Has anyone ever tried to write a formal syntax for all the entries
> > in /proc?   We have bits and pieces of /proc documentation in
> > /usr/src/linux/Documentation, but nothing you could feed directly
> > into a parser generator.  It'd be neat to have a good definition for /proc
> > in the LSB, and have an LSB conformance test that could look in
> > /proc and say "Yup, all the entries there conform to the spec and can
> > be parsed properly."...
> 
> From one point of view (that of the /proc entries...) each file
> is by definition in the proper format. That format is specified
> (in the /proc interface to the driver). Using "proc_printf" is a
> specification for the output.

When two different distributions ship different forks of
the kernel source, which differ in the arguments passed to proc_printf,
which one is right?  
There's no way to tell.  That's why saying "the source is the spec" doesn't cut it.

Also, the source is not a specification a parser generator can use.

A formal spec for /proc entries maintained by e.g. the LSB is needed;
it has to be separate from the source code (to avoid forking problems),
and it should be machine-readable (so we can build parsers from it).

> That DOES NOT mean that no improvements are possible. If the formats
> used by the various modules/drivers has some variation in format from
> access to acess, then the determination of that format must also be
> included. From what I've seen (via "cat /proc/....") the files all
> have a fixed format. Sometimes the number of entries varies, but then
> the count should ALSO be included in the file (in a known place of
> course). The multi-entry files I've looked at (/proc/net) reach the
> EOF to end the list. This is not unreasonable.

Yeah, there's a general style that seems to work; it just needs to be
formalized.
 
> I'm not sure of the usefullness of the title lines that are printed. If
> looked at in raw form, yes the titles are nice. But the utilities
> that are aimed at examining the values should not have to discard them, nor
> should the drivers have to generate them.

I think they're good; they're a little bit like the XML tags you're proposing.
 
> I can live with them anyway, since they are already there.....
> 
> The biggest problem I know of is being able to retrieve structure
> in an atomic manner. Not easy (in any system, not just Linux).

Something SNMP doesn't deal well with, either.  People seem to cope,
though.

- Dan

Re: /proc format (was Device Registry (DevReg) Patch 0.2.0)

[Dan Kegel]

Mark Hahn wrote:
> the main goal at this point is to make kernel proc-related 
> code more efficient, easy-to-use, etc. a purely secondary goal 
> is to make user-space tools more robust, efficient, and simpler. 
> 
> there are three things that need to be communicated through the proc 
> interface, for each chunk of data: its type, it's name and its value. 
> it's critical that data be tagged in some way, since that's the only 
> way to permit back-compatibility. that is, a tool looking for a particular 
> tag will naturally ignore new data with other tags. 

Agreed.

> [three example schemes in use in /proc today]
> I have a sense that all of these could be collapsed into a single 
> api where kernel systems would register hierarchies of tuples of 
> <type,tag,callback>, where callback would be passed the tag, 
> and proc code would take care of "rendering" the data into 
> human readable text (default), binary, or even xml. 

Sounds reasonable to me.  Relieve the modules of having to
format their /proc entries by defining standard code that does
it.   And as an extra bonus, if tuples registration was table-driven,
the tables would define a grammar that could be fed to a parser
generator.

(It sounds a little bit like the snmpd code I'm working on,
actually.  How eerie.)

(It also sounds a little like (gasp) the windows registry,
but hey, that's ok.)

- Dan