Yet another design for /proc. Or actually /kernel.

Yet another design for /proc. Or actually /kernel.

[Erik Hensema]

Here's my go at a new design for /proc. I designed it from a userland
point of view and tried not to drown myself into details.

Design goals
============

- Backward compatibility
	The current /proc interface is coded into many applications.
	A new design must leave room for backward compatibility.
- Forward compatibility
	The new interface must be clearly defined and must not change
	in the future, unless that change is clearly indicated and it
	would cause horrible pain maintaining the current interface.
	(Note: I mean the interface to userland, kernel interfaces may
	change as much as they like)
- Parsable by applications
	Information in /proc isn't really meant for humans to read.
	Maybe programmers, but they aren't human anyway. The new /proc
	interface must be parsable by applications as easily as
	possible.
- Readable by humans
	When there's no compelling need to obscure information, don't.
- As simple as possible
	Don't bloat the interface with all kinds of fancy decorations.
	Let's leave that to userspace.


>From these goals I come to the following design:

Design
======

- Multiple values per file when needed
	A file is a two dimensional array: it has lines and every line
	can consist of multiple fields.
	A good example of this is the current /proc/mounts.
	This can be parsed very easily in all languages.
	No need for single-value files, that's oversimplification.
- No headers, no pretty formatting
	Headers are nice to humans, but applications tend to dislike
	them. Same goes for formatting.
	An example of this is in /proc/partitions or -- horrors --
	/proc/meminfo.
- ASCII values
	ASCII is the only format which can be read by humans, shell
	scripts, perl, C and the rest of the world.
- Maybe we should move out of /proc
	Since this design must leave room for backwards compatibility
	we must assume the current /proc stays around for a while. I
	propose we create a new kernelfs which is to be mounted at
	/kernel.
- Mandatory documentation
	Each and every value must be documented. This documentation
	may be very, very basic, but it must clearly define the entry
	and its intention. This is forward compatibility.


As an example, this is how I would organize the info currently in
/proc/meminfo (any boy, that file is UGLY!):

Every value is in bytes. Rounding to kbytes or mbytes should be done
in userspace.

New		Current entry in /proc/meminfo
--------------- -----------------------------------
mem/total	MemTotal
mem/free	MemFree
mem/shared	MemShared
mem/buffers	Buffers
mem/cached	Cached
mem/active	Active
mem/inactive	Inactive

mem/high/total	HighTotal
mem/high/free	HighFree

mem/low/total	LowTotal
mem/low/free	LowFree

mem/swap/total	SwapTotal
mem/swap/free	SwapFree
mem/swap/cached	SwapCached
mem/swap/files	(current /proc/swaps)
-- 
Erik Hensema (erik@hensema.net)
I'm on the list, no need to Cc: me, though I appreciate one if your
mailer doesn't support the References header.

Re: Yet another design for /proc. Or actually /kernel.

[Alan Cox]

> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

Did you have to change the subject line. It makes it harder to kill file 
when people keep doing that

/proc discussions

[Tim Jansen]

On Wednesday 07 November 2001 20:27, Alan Cox wrote:
> > Here's my go at a new design for /proc. I designed it from a userland
> > point of view and tried not to drown myself into details.
> Did you have to change the subject line. It makes it harder to kill file
> when people keep doing that

There is an easy way for you, or even better, Linus to stop these discussions:

Just say, in unambigous words, what kind of patch you would accept, if any. 
The open questions are: in which format should data (from driver and kernel) 
be exported? How can the kernel, drivers and devices be configured from user 
space?
Even better, you could also say a few words about the possible disadvantages 
that the decision has and why we shouldn't care. Something like "performance 
is irrelevant", "it doesn't matter if the format cannot be changed without 
breaking user applications", "for every format there must be a strict 
definition of how to parse it so that the new fields can be added later", "we 
don't need to define the format as long at it is readable for human readers", 
"if you need a pretty format write some user space tool that formats it", "it 
doesnt matter if each file uses a completely different format", "it is not 
acceptable to break any user space applications before 2.x", "strict typing 
is irrelevant"...


I REALLY do have problems to define any proc interface without having at 
least one of the disadvantages above. I have spent some time to find LT 
statements about proc/sysctl in order to find out which way may have a chance 
to get into the kernel (this time is possibly better spent than writing 
another proc replacement :). Maybe I have missed some important ones, in 
essence they are:

4 Nov 2001:
"In short: /proc is ASCII, and will so remain while I maintain a kernel.
Anything else is stupid.
Handling spaces and newlines is easy enough - see the patches from Al
Viro, for example."

4 Nov 2000:
"[context added]
>        create_proc_entries(NULL,
>                "test:{bar:{x:%d,y:%d,z:%d},foo:%f}",
>                &x, &y, &z, foo_fun);
>
> creates a "/proc/test" directory, which further contains a
> subdirectory "bar" and a file "foo". The "bar" subdirectory contains
> three files "x", "y" and "z".
> 
> The formatting argument "%d" takes a pointer to an integer. When
> reading such a file (in this case "x", "y", or "z"), the value is
> shown as ascii. Writing to the file (again in ascii) updates the
> value. The "%f" formatting argument allows you to pass an arbitrary
> user function for generating output. Clearly, there are potentially
> quite a number of standard/useful formatting arguments.
> 
> I've done a quick, dirty, unfinished implementation of this idea, so
> people can get the picture. Attached.
> 
> Many people will hate this because (1) it's doing parsing within the
> kernel, (2) it tends to favour ascii I/O, (3) it tends to favour deep
> directory hierarchies,(4) it uses recursion :-)

I like the idea a lot: it's simple, it's clean, and it does make it truly
trivial to build up a simple /proc architecture.
"

7 Jan 2000:
"In contrast, sysctl isn't all that performance-sensitive, AND they are
extremely hierarchical, AND they depend on configuration and timing. 

In short, sysctl NEEDS:
 - "naming": you cannot name the sysctl space with a number: it is much
   too dynamic for that. How do you enumerate drivers? Give them random
   numbers?
 - "listing": showing which sysctl's are there, in a hierarchical manner.
   Again, a listing is useless with a number.
 - "hierarchy". You have different devices, but they have the same
   controls. Do they get the same name? Yes. But in different places in
   the hierarchy.

In short, you NEED a filesystem. You need to be able to "ls" the thing.
You need to be able to search the thing. You need to be doing all the
things you can do with a real filesystem."

6 Jan 2000:
"The thing to do is to create a

        /proc/drivers/<drivername>/ 

directory. The /proc/drivers/ directory is already there, so you'd
basically do something like

        create_proc_info_entry("driver/mydriver/status", 0, NULL, 
mydriver_status_read);
"

1 Jan 1998:
"- I think it should be a separate filesystem type. It was a mistake in
 the first place to overload /proc as much as we did, we shouldn't
 continue that (I should really have split out the "per-process" things
 and the "global" things to two separate filesystems). "


Is the 4-Nov-2000 mail combined with 1-Jan-1998 the answer?

bye...

Re: /proc discussions

[Alexander Viro]

On Thu, 8 Nov 2001, Tim Jansen wrote:

> There is an easy way for you, or even better, Linus to stop these discussions:
> 
> Just say, in unambigous words, what kind of patch you would accept, if any. 

.procmailrc one would do nicely.

Re: Yet another design for /proc. Or actually /kernel.

[Daniel R. Warner]

Erik Hensema wrote:

> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

<snip>

> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.


> 	No need for single-value files, that's oversimplification.
<snip>


This is fine for reading, but it makes it harder for humans to change 
values in /proc - eg, echo 0 > /proc/sys/net/ipv4/tcp_ecn

-D

Re: Yet another design for /proc. Or actually /kernel.

[Allen Campbell]

On Wed, Nov 07, 2001 at 02:42:48PM -0500, Daniel R. Warner wrote:
> Erik Hensema wrote:
> 
> > - Multiple values per file when needed
> > 	A file is a two dimensional array: it has lines and every line
> > 	can consist of multiple fields.
> > 	A good example of this is the current /proc/mounts.
> > 	This can be parsed very easily in all languages.
> 
> 
> > 	No need for single-value files, that's oversimplification.
> <snip>
> 
> 
> This is fine for reading, but it makes it harder for humans to change 
> values in /proc - eg, echo 0 > /proc/sys/net/ipv4/tcp_ecn
> 

'Multiple value' files can be made easy to 'write'.  The only
requirement is each 'field' in the file have a unique label.  Then
it's a common associative array, requiring some generic filesystem
write magic to handle the input:

echo "label:1" > /proc/...

The 'generic write magic' would require (at least, without even
more magic) that all /proc files conform to the schema.  This is
probably a _good_ thing.

-- 
  Allen Campbell       |  Lurking at the bottom of the
  allenc@verinet.com   |   gravity well, getting old.

Re: Yet another design for /proc. Or actually /kernel.

[H. Peter Anvin]

Followup to:  <slrn9uj1nf.5lj.spamtrap@dexter.hensema.xs4all.nl>
By author:    spamtrap@use.reply-to (Erik Hensema)
In newsgroup: linux.dev.kernel
> 
> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.
> 	No need for single-value files, that's oversimplification.
> 

Actually, /proc/mounts is currently broken, and is an excellent
example of why the above statement simply isn't true unless you apply
another level of indirection: try mounting something on a directory
the name of which contains whitespace in any form (remember, depending
on your setup this may be doable by an unprivileged user...)

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Yet another design for /proc. Or actually /kernel.

[Justin A]

Ahh yes I've seen that happen.  It's not too common in standard unix
practices, but comes up a lot when mounting windows shares.  Mounting
"//bobs computer/my folder with stuff in it" does not have good results.
Only way I found to fix that was to umount -a -t smbfs.

-Justin

On Wed, Nov 07, 2001 at 12:58:24PM -0800, H. Peter Anvin wrote:
> Followup to:  <slrn9uj1nf.5lj.spamtrap@dexter.hensema.xs4all.nl>
> By author:    spamtrap@use.reply-to (Erik Hensema)
> In newsgroup: linux.dev.kernel
> > 
> > - Multiple values per file when needed
> > 	A file is a two dimensional array: it has lines and every line
> > 	can consist of multiple fields.
> > 	A good example of this is the current /proc/mounts.
> > 	This can be parsed very easily in all languages.
> > 	No need for single-value files, that's oversimplification.
> > 
> 
> Actually, /proc/mounts is currently broken, and is an excellent
> example of why the above statement simply isn't true unless you apply
> another level of indirection: try mounting something on a directory
> the name of which contains whitespace in any form (remember, depending
> on your setup this may be doable by an unprivileged user...)
> 
> 	-hpa
> -- 
> <hpa@transmeta.com> at work, <hpa@zytor.com> in private!
> "Unix gives you enough rope to shoot yourself in the foot."
> http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Yet another design for /proc. Or actually /kernel.

[Rusty Russell]

On 7 Nov 2001 19:09:35 GMT
spamtrap@use.reply-to (Erik Hensema) wrote:

> 
> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.
> 	No need for single-value files, that's oversimplification.

No, it deals nicely with any possible values in the file.  And without headers,
how do I know what's what?  And how do I update one value.

Meanwhile, there's far too much talk, far too little code.  Will post new patch
next week.

Rusty.

Re: Yet another design for /proc. Or actually /kernel.

[Stephen Satchell]

At 12:58 PM 11/7/01 -0800, H. Peter Anvin wrote:
>Actually, /proc/mounts is currently broken, and is an excellent
>example of why the above statement simply isn't true unless you apply
>another level of indirection: try mounting something on a directory
>the name of which contains whitespace in any form (remember, depending
>on your setup this may be doable by an unprivileged user...)

Good man.

Now you know why I insisted that ALL STRINGS have a standard escape system 
in my less-than-strawman proposal.

You'll find other places in /proc where white space is not necessarily a 
delimiter.  (Too lazy to go find them right now, however.)

Satch

Re: Yet another design for /proc. Or actually /kernel.

[William Knop]

>>Here's my go at a new design for /proc. I designed it from a userland
>>point of view and tried not to drown myself into details.
>
>Did you have to change the subject line. It makes it harder to kill >file 
>when people keep doing that

You really consider this discussion to be unimportant? Granted, the lack of 
organization in /proc is not a bug, but keeping the kernel organized is part 
of maintenence. Like cleaning my room-- it's not a showstopper if it is 
messy, but it is nicer and easier to work in when neat.

Will Knop
w_knop@hotmail.com

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

Re: Yet another design for /proc. Or actually /kernel.

[John Levon]

On Wed, Nov 07, 2001 at 02:41:26PM -0500, William Knop wrote:

> You really consider this discussion to be unimportant? Granted, the lack of 
> organization in /proc is not a bug, but keeping the kernel organized is part 
> of maintenence. Like cleaning my room-- it's not a showstopper if it is 
> messy, but it is nicer and easier to work in when neat.

Like cleaning your room, talking endlessly about how doesn't get you anywhere.
Show some code ...

john

-- 
"This bulletin discusses three security vulnerabilities that are unrelated
except in the sense that both affect ISA Server 2000"
	- Microsoft Product Security

Re: Yet another design for /proc. Or actually /kernel.

[Erik Hensema]

John Levon (moz@compsoc.man.ac.uk) wrote:
>On Wed, Nov 07, 2001 at 02:41:26PM -0500, William Knop wrote:
>
>> You really consider this discussion to be unimportant? Granted, the lack of 
>> organization in /proc is not a bug, but keeping the kernel organized is part 
>> of maintenence. Like cleaning my room-- it's not a showstopper if it is 
>> messy, but it is nicer and easier to work in when neat.
>
>Like cleaning your room, talking endlessly about how doesn't get you anywhere.
>Show some code ...

Just starting to code without giving much thought into it give us... the
current /proc. It may be a good idea to actually design the thing this
time.

-- 
Erik Hensema (erik@hensema.net)
I'm on the list, no need to Cc: me, though I appreciate one if your
mailer doesn't support the References header.

Re: Yet another design for /proc. Or actually /kernel.

[Remco Post]

> >>Here's my go at a new design for /proc. I designed it from a userland
> >>point of view and tried not to drown myself into details.
> >
> >Did you have to change the subject line. It makes it harder to kill >file 
> >when people keep doing that
> 
> You really consider this discussion to be unimportant? Granted, the lack of 
> organization in /proc is not a bug, but keeping the kernel organized is part 
> of maintenence. Like cleaning my room-- it's not a showstopper if it is 
> messy, but it is nicer and easier to work in when neat.
> 
> Will Knop
> w_knop@hotmail.com

Hi,

the discussion is irrelevant. Despite what everybody thinks, Linus thinks 
/proc must be not binary, so it will stay that way for those of us who run 
Linus kernels...

I can inmagine people like Alan ignoring this discussion after such a 
statement, the outcome of the discussion is irrelevant for the kernel 
development.


-- 
Met vriendelijke groeten,

Remco Post

SARA - Stichting Academisch Rekencentrum Amsterdam
High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668 3167

"I really didn't foresee the Internet. But then, neither did the computer
industry. Not that that tells us very much of course - the computer industry
didn't even foresee that the century was going to end." -- Douglas Adams

Re: Yet another design for /proc. Or actually /kernel.

[Ricky Beam]

>the discussion is irrelevant. Despite what everybody thinks, Linus thinks
>/proc must be not binary, so it will stay that way for those of us who run
>Linus kernels...

Linus has been "wrong" before.  It will require good code and numbers
backing that codes "goodness" before Linus will begin to listen.  Yes,
a new procfs format will break a great deal of userland toys, so the
changes had better be worth it and sufficient to never, EVER require
a complete overhaul in the future.

>I can inmagine people like Alan ignoring this discussion after such a
>statement, the outcome of the discussion is irrelevant for the kernel
>development.

I think Alan is already ignoring the entire discussion.

--Ricky

Re: Yet another design for /proc. Or actually /kernel.

[Horst von Brand]

Ricky Beam <jfbeam@bluetopia.net> said:
> >the discussion is irrelevant. Despite what everybody thinks, Linus thinks
> >/proc must be not binary, so it will stay that way for those of us who run
> >Linus kernels...
> 
> Linus has been "wrong" before.  It will require good code and numbers
> backing that codes "goodness" before Linus will begin to listen.  Yes,
> a new procfs format will break a great deal of userland toys, so the
> changes had better be worth it and sufficient to never, EVER require
> a complete overhaul in the future.

/proc for process info is a given (many Unices have it, it is nice at least
for compatibility).

/proc for random other garbage should go away. To get at some value you can
get via specialized calls by read(2) also is just kernel bloat.
-- 
Dr. Horst H. von Brand                Usuario #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: Yet another design for /proc. Or actually /kernel.

[Martin Dalecki]

Horst von Brand wrote:
> 
> Ricky Beam <jfbeam@bluetopia.net> said:
> > >the discussion is irrelevant. Despite what everybody thinks, Linus thinks
> > >/proc must be not binary, so it will stay that way for those of us who run
> > >Linus kernels...
> >
> > Linus has been "wrong" before.  It will require good code and numbers
> > backing that codes "goodness" before Linus will begin to listen.  Yes,
> > a new procfs format will break a great deal of userland toys, so the
> > changes had better be worth it and sufficient to never, EVER require
> > a complete overhaul in the future.
> 
> /proc for process info is a given (many Unices have it, it is nice at least
> for compatibility).
> 
> /proc for random other garbage should go away. To get at some value you can
> get via specialized calls by read(2) also is just kernel bloat.

My absolute favourite is the following in arch/i386/kernel/irq.c!!!!

Disease symptoms like this are distributed all over the kernel code.

========================================================================
static struct proc_dir_entry * root_irq_dir;
static struct proc_dir_entry * irq_dir [NR_IRQS];

#define HEX_DIGITS 8

static unsigned int parse_hex_value (const char *buffer,
		unsigned long count, unsigned long *ret)
{
	unsigned char hexnum [HEX_DIGITS];
	unsigned long value;
	int i;

RE: Yet another design for /proc. Or actually /kernel.

[Brenneke, Matthew Jeffrey (UMR-Student)]

>-----Original Message-----
>From: H. Peter Anvin [mailto:hpa@zytor.com]
>Sent: Wednesday, November 07, 2001 2:58 PM
>To: linux-kernel@vger.kernel.org
>Subject: Re: Yet another design for /proc. Or actually /kernel.


>Followup to:  <slrn9uj1nf.5lj.spamtrap@dexter.hensema.xs4all.nl>
>By author:    spamtrap@use.reply-to (Erik Hensema)
>In newsgroup: linux.dev.kernel
>> 
>> - Multiple values per file when needed
>> 	A file is a two dimensional array: it has lines and every line
>> 	can consist of multiple fields.
>> 	A good example of this is the current /proc/mounts.
>> 	This can be parsed very easily in all languages.
>> 	No need for single-value files, that's oversimplification.
>> 

>Actually, /proc/mounts is currently broken, and is an excellent
>example of why the above statement simply isn't true unless you apply
>another level of indirection: try mounting something on a directory
>the name of which contains whitespace in any form (remember, depending
>on your setup this may be doable by an unprivileged user...)

>	-hpa


mbrennek@spaceheater:/home/mbrennek# mkdir stuff\ and
mbrennek@spaceheater:/home/mbrennek# mount -t vfat /dev/hda1
/home/mbrennek/stuff\ and/
mbrennek@spaceheater:/home/mbrennek# cat /proc/mounts
/dev/ide/host0/bus0/target1/lun0/part1 / reiserfs rw 0 0
/dev/hdb2 /home reisferfs rw 0 0
none /dev/pts devpts rw 0 0
non /proc proc rw 0 0
/dev/hda5 /mnt/stuff vfat rw,nosuid,nodev,noexec 0 0
/dev/hda1 /home/mbrennek/stuff\040and vfat rw 0 0
mbrennek@spaceheater:/home/mbrennek#

Are you refering to the 040?

-Matt

Re: Yet another design for /proc. Or actually /kernel.

[H. Peter Anvin]

Brenneke, Matthew Jeffrey (UMR-Student) wrote:

> 
>>Actually, /proc/mounts is currently broken, and is an excellent
>>example of why the above statement simply isn't true unless you apply
>>another level of indirection: try mounting something on a directory
>>the name of which contains whitespace in any form (remember, depending
>>on your setup this may be doable by an unprivileged user...)
>>
> 
>>	-hpa
>>
> 
> 
> mbrennek@spaceheater:/home/mbrennek# mkdir stuff\ and
> mbrennek@spaceheater:/home/mbrennek# mount -t vfat /dev/hda1
> /home/mbrennek/stuff\ and/
> mbrennek@spaceheater:/home/mbrennek# cat /proc/mounts
> /dev/ide/host0/bus0/target1/lun0/part1 / reiserfs rw 0 0
> /dev/hdb2 /home reisferfs rw 0 0
> none /dev/pts devpts rw 0 0
> non /proc proc rw 0 0
> /dev/hda5 /mnt/stuff vfat rw,nosuid,nodev,noexec 0 0
> /dev/hda1 /home/mbrennek/stuff\040and vfat rw 0 0
> mbrennek@spaceheater:/home/mbrennek#
> 
> Are you refering to the 040?
> 


Right, a good example of "additional encapsulation".

	-hpa

Re: Yet another design for /proc. Or actually /kernel.

[antirez]

On Wed, Nov 07, 2001 at 03:13:25PM -0600, Brenneke, Matthew Jeffrey (UMR-Student) wrote:
> /dev/hda1 /home/mbrennek/stuff\040and vfat rw 0 0
[snip]
> 
> Are you refering to the 040?

This works but, if /proc will really be replaced, another
idea can be to organize the stuff to get something of more
coherent than:

value1a value1b value1c
value2a value2b value2c

that's more human readable than machine parsable.
Something that can fix both the problems (quoting and format) is
the following:

put every string inside () and only quotes ( and ) with \
and quotes \ itself with \\, than use brackets to delimit
string _and_  provide information about the format:

((dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
((/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))

and so on. With a simple parser you get a safe delimiter
for a single element and you know that there are two "objects"
of 6 elements.

An extension may be to use the first row to describe the
elements inside every object. After this the output of
/proc/mounts will be:

((device)(mountpoint)(type)(option)(dump)(fsck))
((dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
((/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))

Good userspace software will never fail if you add more information.
If you need to change the format of something it's just possible
to add a new field and leave the old one for some time for backward
compatibility.

With some simple extension you can add comments or the ability
to redefine the fileds description for the next elements.
With the last improvments it looks like:

((type)(field0)(field1)...(fieldN))

/proc/mounts will be:

((rem)(mounted filesystems))
((format)(device)(mountpoint)(type)(option)(dump)(fsck))
((data)(dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
((data)(/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))
((rem)(number of filesystems mounted))
((format)(mount_count))
((data)(2))

Last fix can be to surround all this with (), to detect
truncated outputs, since it seems that often the /proc outputs
can only use one page.

It looks a bit too complex, but actually it's very simple to
generate and parse. Expecially if someone release a "standard"
userspace library to parse it.

I'm sure all here already thought at something like it, and
probably there is a good reason to do it different, sorry if I
missed the point.

regards,
Salvatore

Re: Yet another design for /proc. Or actually /kernel.

[H. Peter Anvin]

antirez wrote:

> On Wed, Nov 07, 2001 at 03:13:25PM -0600, Brenneke, Matthew Jeffrey (UMR-Student) wrote:
> 
>>/dev/hda1 /home/mbrennek/stuff\040and vfat rw 0 0
>>
> [snip]
> 
>>Are you refering to the 040?
>>
> 
> This works but, if /proc will really be replaced, another
> idea can be to organize the stuff to get something of more
> coherent than:
> 
> value1a value1b value1c
> value2a value2b value2c
> 
> that's more human readable than machine parsable.
> Something that can fix both the problems (quoting and format) is
> the following:
> 
> put every string inside () and only quotes ( and ) with \
> and quotes \ itself with \\, than use brackets to delimit
> string _and_  provide information about the format:
> 
> ((dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
> ((/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))
> 
> and so on. With a simple parser you get a safe delimiter
> for a single element and you know that there are two "objects"
> of 6 elements.
> 


You still need quoting, so why on earth does this make it easier to parse?
 It doesn't; it just makes it harder to read for humans.

	-hpa

Re: Yet another design for /proc. Or actually /kernel.

[David Ford]

That doesn't solve anything if the data value includes ( or ).  It just 
avoids ' ' in the data value and adds complexity.

-d

antirez wrote:

>((dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
>((/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))
>

Re: Yet another design for /proc. Or actually /kernel.

[antirez]

On Wed, Nov 07, 2001 at 07:54:21PM -0500, David Ford wrote:
> That doesn't solve anything if the data value includes ( or ).  It just 
> avoids ' ' in the data value and adds complexity.

Wrong, exampel of () in data:

((data)(\(\)))

About the complexity. It only "looks" complex. But from the
machine point of view it's very simple to parse.
Note that the strong advantage of this isn't the quoting,
you can quote anyway in 1000 different ways. The advantage
is that data is structured and parsing does not rely on
spaces or newlines, but just on ().
With this syntax you can express data as complex and structured
as you want but the parsing is still simple.

Regards,
Salvatore

Re: Yet another design for /proc. Or actually /kernel.

[H. Peter Anvin]

antirez wrote:

> On Wed, Nov 07, 2001 at 07:54:21PM -0500, David Ford wrote:
> 
>>That doesn't solve anything if the data value includes ( or ).  It just 
>>avoids ' ' in the data value and adds complexity.
>>
> 
> Wrong, exampel of () in data:
> 
> ((data)(\(\)))
> 
> About the complexity. It only "looks" complex. But from the
> machine point of view it's very simple to parse.
> Note that the strong advantage of this isn't the quoting,
> you can quote anyway in 1000 different ways. The advantage
> is that data is structured and parsing does not rely on
> spaces or newlines, but just on ().
> With this syntax you can express data as complex and structured
> as you want but the parsing is still simple.
> 


You just changed spaces and newlines to ( and ) -- it doesn't really solve
anything unless you want three levels of nesting or more; in which case
you have *WAY* too much data in a single proc item.

	-hpa

Re: Yet another design for /proc. Or actually /kernel.

[antirez]

On Wed, Nov 07, 2001 at 05:26:32PM -0800, H. Peter Anvin wrote:
> > About the complexity. It only "looks" complex. But from the
> > machine point of view it's very simple to parse.
> > Note that the strong advantage of this isn't the quoting,
> > you can quote anyway in 1000 different ways. The advantage
> > is that data is structured and parsing does not rely on
> > spaces or newlines, but just on ().
> > With this syntax you can express data as complex and structured
> > as you want but the parsing is still simple.
> > 
> 
> 
> You just changed spaces and newlines to ( and ) -- it doesn't really solve
> anything unless you want three levels of nesting or more; in which case
> you have *WAY* too much data in a single proc item.
> 
> 	-hpa

There are anyway different ways to output the same data, and yes,
probably spaces/tabs/newlines are more human readable, but I think
the right solution isn't something that limits a-priori the
complexity of the output. This will make developers more prone
to invent their own formats for special stuff. the lisp-like way
allows you to automagically put a description of the format with
little efforts, simple parsing, unlimited complexity.
Maybe you want limited complexity, but the format isn't your limit
anyway.

About the two level of nesting, take a look at /proc/net/netstat.
it's not very clear, but in lisp-like it can be translated to:

((TcpExt)((SyncookiesSent)(0)))

and so on. For every kind of proc output you can find today, there
is a good way to convert it in that format, that is at the same
time used by all the entries. I think you will hardly get the same
with space/tabs/newlines without to indirectly use it like (), that
will probably result in something of more complex to generate/parse.

I can't see any strong reason to adopt a format that will for sure
fail at some time in the future.

BTW I see that the idea isn't well accepted, so I'll be quiet ;)

Regards,
Salvatore

Re: Yet another design for /proc. Or actually /kernel.

[Stephen Satchell]

At 01:20 AM 11/8/01 +0100, antirez wrote:
>/proc/mounts will be:
>
>((rem)(mounted filesystems))
>((format)(device)(mountpoint)(type)(option)(dump)(fsck))
>((data)(dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
>((data)(/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))
>((rem)(number of filesystems mounted))
>((format)(mount_count))
>((data)(2))

Looks a little like LISPTH.  Flex and Bison to the rescue!

(signed) anonymous

Re: Yet another design for /proc. Or actually /kernel.

[antirez]

On Wed, Nov 07, 2001 at 04:44:50PM -0800, Stephen Satchell wrote:
> At 01:20 AM 11/8/01 +0100, antirez wrote:
> >/proc/mounts will be:
> >
> >((rem)(mounted filesystems))
> >((format)(device)(mountpoint)(type)(option)(dump)(fsck))
> >((data)(dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
> >((data)(/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))
> >((rem)(number of filesystems mounted))
> >((format)(mount_count))
> >((data)(2))
> 
> Looks a little like LISPTH.  Flex and Bison to the rescue!
> 
> (signed) anonymous

I'm really a newbie. Maybe I missed the whole point but
stay sure you don't need flex and bison to parse this.
It's _really_ simple, you need little code of little
complexity, to get something of flexible that can be
used for long time.

If you don't trust me I can write a little C library to parse
this just for provide a proof.
This isn't only simple to parse, but even simple to generate,
that's an important point.

regards,
Salvatore

RE: Yet another design for /proc. Or actually /kernel.

[Jonathan Lundell]

At 3:13 PM -0600 11/7/01, Brenneke, Matthew Jeffrey (UMR-Student) wrote:
>  >> - Multiple values per file when needed
>>>	A file is a two dimensional array: it has lines and every line
>>>	can consist of multiple fields.
>>>	A good example of this is the current /proc/mounts.
>>>	This can be parsed very easily in all languages.
>>>	No need for single-value files, that's oversimplification.
>>>
>
>>Actually, /proc/mounts is currently broken, and is an excellent
>>example of why the above statement simply isn't true unless you apply
>>another level of indirection: try mounting something on a directory
>>the name of which contains whitespace in any form (remember, depending
>>on your setup this may be doable by an unprivileged user...)

If [tag plus] multiple values are allowed on a line, the field 
separation needs to be unambiguous. So quoting/escaping is required 
in some cases. Spaces are common enough in a value that white space 
maybe doesn't make a very good field separator.

Or just quote all strings (and escape quotes). Interpret values as C 
would, 0x for hex, "... for strings, and so on for floating point, 
octal, &c. Heck, you could even have typing (3UL) or casts, though 
that's probably going too far.
-- 
/Jonathan Lundell.

Re: Yet another design for /proc. Or actually /kernel.

[Stuart Young]

At 01:20 AM 8/11/01 +0100, antirez wrote:
>put every string inside () and only quotes ( and ) with \
>and quotes \ itself with \\, than use brackets to delimit
>string _and_  provide information about the format:
>
>((dev/hda1)(/home/mbrennek/stuff and)(vfat)(rw)(0)(0))
>((/dev/hda2)(/var/tmp)(ext2)(rw)(0)(0))

Now that is just really annoying, and hardly useful.

Why not just allow the user to specify what the field separation character 
is. If the entry/file is opened for writing, then the data passed to it is 
used as the separator. If it's opened in read, just output as normal.

So if say /proc/kernel/foo normally exports...

bar-a1  bar-a2  bar-a3
bar-b1  bar-b2  bar-b3
[===>         ] 40%

...and you pass a colon ":", it spits out...

bar-a1:bar-a2:bar-a3
bar-b1:bar-b2:bar-b3
40

This allows the user to make the choice. If they use the wrong character, 
they can always change it. If they don't specify one, they get the default 
output (which is what existing scripts will expect). From a code 
perspective, you should only need to make all your separators (formatting) 
as variables in the code, and substitute the separator character(s) if 
there is one that's been passed, and just drop the remaining 'pretty' 
garbage totally.

Also means you can easily use command line tools like cut (-d option), 
readline (the read function in bash, and the $IFS variable), and so on, to 
parse the data, quickly and effectively.


AMC Enterprises P/L    - Stuart Young
First Floor            - Network and Systems Admin
3 Chesterville Rd      - sgy@amc.com.au
Cheltenham Vic 3192    - Ph:  (03) 9584-2700
http://www.amc.com.au/ - Fax: (03) 9584-2755