Scripting support

Scripting support

[Marco Gerards]

Hi,

Last days I have been thinking about scripting support.  With my last
patch it should be easy to implement.  I hope Vladimir is still
interested in this.

The most important is the general design.  I think we shouldn't put
too much in the core and not even in normal mode.  This is the same
approach bash has taken.  I think the core functionality should be:

- variables (we have that)
- functions
    Functions should work *exactly* like normal GRUB commands.  In
    that case we can easily mix GRUB commands and functions.
- Return values
    All commands should be able to return a value. That's also what
    bash has. This value can be stored in a variable or simply
    returned, I am not too sure about that.  Just check what bash
    does.
- Conditional statements like if and case
- loops: until, while and for

Whenever a command line should be executed or when a script file is
executed, it's the same.  The block of code is just parsed and
executed by a function (the parser, I guess :)).

It's important to see that GRUB does not do anything like
calculations.  All this can be moved into a command.  When making a
bash script you have the same, there is the expr command.  So we have
to keep the core functionality minimal and should offer the commands
in command/ like we do now.

First the lexer.  I think it should work like this:

normal/parser.c should contain the lexer.  The prototypes etc. can
just be added to include/grub/parser.h.

There are just a few tokens we have to recognize.  Think about stuff
like a string of text, if, elif, case, ;, [], etc.

The parser can be quite simple.  It has to support all kinds of loops,
etc.  I am not sure how to deal with that.  perhaps we have to feed
the block within a for loop (for example) to the parser until the for
loop exits.  Another possibility is creating some kind of P-code and
parse that later on.  I wonder what the best approach is.

For the parser we should use bison, I think.  Writing a handwritten
parser is acceptable if some standard approach is taken and it's
really really clean and well documented.

Any idea is welcome.  I hope we can discuss the implementation in
detail so we can have scripting support soon.

Thanks,
Marco

Re: Scripting support

[Vladimir Serbinenko]

Marco Gerards wrote:

>Hi,
>
>Last days I have been thinking about scripting support.  With my last
>patch it should be easy to implement.  I hope Vladimir is still
>interested in this.
>  
>
Yes. Exactly now I'm on holiday so I have a lot of time. And also I have 
some ideas for some new GRUB features but theese features aren't useful 
without scripting so I'm interested more than ever.

>The most important is the general design.  I think we shouldn't put
>too much in the core and not even in normal mode.  This is the same
>approach bash has taken.  I think the core functionality should be:
>
>- variables (we have that)
>- functions
>    Functions should work *exactly* like normal GRUB commands.  In
>    that case we can easily mix GRUB commands and functions.
>  
>
In bash when you define a function it appears under environment 
variables. IMHO it's useless to store completely the source code as an 
environment variables. I propose that the functions will be preparsed 
(converted to internal format ready for execution) because it will 
gretelya simplify lexer and parser

>- Return values
>    All commands should be able to return a value. That's also what
>    bash has. This value can be stored in a variable or simply
>    returned, I am not too sure about that.  Just check what bash
>    does.
>  
>
Could it be merged with error variable. It's what bash does

>- Conditional statements like if and case
>- loops: until, while and for
>
>  
>
I propose the following argorithm. Bison parser transforms the text-form 
to internal form of a linked list of commands with some special commands 
like jump if last value was true, ... . Think about assembler. Then it's 
executed by simple engine that works like CPU: it takes one command from 
a linked list. Executes it and changes the pointer so it points to the 
next command. A command is a structure containing:
    variable showing it's type (normal command, jump, loop, end of loop, 
...)
    union of data necessary for necessary commands
    pointer to next command.

I made a simple example I posted as preview version of scripting engine 
and I also posted on this list simple design doc.
   
Any ideas are welcome.
   
                                                                         
                                                      Vladimir 'phcoder' 
Serbinenko

Re: Scripting support

[Marco Gerards]

Vladimir Serbinenko <phcoder@gmail.com> writes:

Hi Vladimir,

>>Last days I have been thinking about scripting support.  With my last
>>patch it should be easy to implement.  I hope Vladimir is still
>>interested in this.
>>
>>
> Yes. Exactly now I'm on holiday so I have a lot of time. And also I
> have some ideas for some new GRUB features but theese features aren't
> useful without scripting so I'm interested more than ever.

Nice.

>>The most important is the general design.  I think we shouldn't put
>>too much in the core and not even in normal mode.  This is the same
>>approach bash has taken.  I think the core functionality should be:
>>
>>- variables (we have that)
>>- functions
>>    Functions should work *exactly* like normal GRUB commands.  In
>>    that case we can easily mix GRUB commands and functions.
>>
>>
> In bash when you define a function it appears under environment
> variables. IMHO it's useless to store completely the source code as an
> environment variables. I propose that the functions will be preparsed
> (converted to internal format ready for execution) because it will
> gretelya simplify lexer and parser

Right.  We should discuss the internal format on the list, I think.

>>- Return values
>>    All commands should be able to return a value. That's also what
>>    bash has. This value can be stored in a variable or simply
>>    returned, I am not too sure about that.  Just check what bash
>>    does.
>>
>>
> Could it be merged with error variable. It's what bash does

Ok.

>>- Conditional statements like if and case
>>- loops: until, while and for
>>
>>
>>
> I propose the following argorithm. Bison parser transforms the
> text-form to internal form of a linked list of commands with some
> special commands like jump if last value was true, ... . Think about
> assembler. Then it's executed by simple engine that works like CPU: it
> takes one command from a linked list. Executes it and changes the
> pointer so it points to the next command. A command is a structure
> containing:
>     variable showing it's type (normal command, jump, loop, end of
> loop, ...)
>     union of data necessary for necessary commands
>     pointer to next command.

Right.  And this can be kept quite simple.  The commands itself are
just a string of text.  The loops just check a variable, no?  It can
be something like:

for foo in 1 2 3
 do
   echo $i
 done

That can be translated into some kind of pseudo language:

1: list = 1 2 3
2: read i, list
3: echo $i
4: check list
5: je 2

I know it's stupid and kind of silly like this.  But we have to define
a language.  It's important to discuss this on the list, IMO.

> I made a simple example I posted as preview version of scripting
> engine and I also posted on this list simple design doc.
>    Any ideas are welcome.

I think I gave you some ideas. ;)

Which mail are you talking about (subject and date)?  I must have
missed a few the last months. :-(

--
Marco

Re: Scripting support

[Vladimir Serbinenko]

Hello

>
>  
>
>>In bash when you define a function it appears under environment
>>variables. IMHO it's useless to store completely the source code as an
>>environment variables. I propose that the functions will be preparsed
>>(converted to internal format ready for execution) because it will
>>gretelya simplify lexer and parser
>>    
>>
>
>Right.  We should discuss the internal format on the list, I think.
>
>  
>
>Right.  And this can be kept quite simple.  The commands itself are
>just a string of text.  The loops just check a variable, no?  It can
>be something like:
>
>for foo in 1 2 3
> do
>   echo $i
> done
>
>That can be translated into some kind of pseudo language:
>
>1: list = 1 2 3
>2: read i, list
>3: echo $i
>4: check list
>5: je 2
>
>I know it's stupid and kind of silly like this.  But we have to define
>a language.  It's important to discuss this on the list, IMO.
>
>  
>
I thought about (a):
1: command=forin variable=i list=1 2 3 end=NULL /* it will be 4 if 
script continues*/
2: echo $i
3: jump 1
Or (b):
1:splitpush 1 2 3
2:pop list
3:read i, list
4:push list
5:echo $i
6:pop list
7:check list
8:push list
9:je 2
10:pop list
The question is how many commands to make: a lot (like in a where nearly 
every internal command corresponds to a scripting command) or make so 
few commands as possible. I wonder which approach is better (it's 
question about bugs and readibility, of course, not about speed)


>Which mail are you talking about (subject and date)?  I must have
>missed a few the last months. :-(
>  
>
Re: [Patch] Scripting engine 25.09.2005
[Patch] Scripting engine 24.08.2005
It's just demonstration of some ideas. Now I think with new splitter it 
would be better to keep arglist unparsed. In this patch it was that if 
we have
echo 1 2 3\;\$ ab$i;
lexer returns (schematically)
STRING(echo) STRING(1) STRING(2) STRING (3;$) STRING(abVAR(i)) SEMICOLON
now I would do that it returns:
STRING(echo 1 2 3\;\$ ab$i) SEMICOLON


                                                                         
                                          Vladimir 'phcoder' Serbinenko

Re: Scripting support

[Marco Gerards]

Vladimir Serbinenko <phcoder@gmail.com> writes:

>>That can be translated into some kind of pseudo language:
>>
>>1: list = 1 2 3
>>2: read i, list
>>3: echo $i
>>4: check list
>>5: je 2
>>
>>I know it's stupid and kind of silly like this.  But we have to define
>>a language.  It's important to discuss this on the list, IMO.
>>
>>
>>
> I thought about (a):

Right.  It's also possible.

> The question is how many commands to make: a lot (like in a where
> nearly every internal command corresponds to a scripting command) or
> make so few commands as possible. I wonder which approach is better
> (it's question about bugs and readibility, of course, not about speed)

I think we first have to determine:

1) Which commands should be supported.
2) The smallest subset of commands that should be generated.

Did you have a look at how bash does this?  Perhaps it even has a
completely different approach...

>>Which mail are you talking about (subject and date)?  I must have
>>missed a few the last months. :-(
>>
>>
> Re: [Patch] Scripting engine 25.09.2005
> [Patch] Scripting engine 24.08.2005
> It's just demonstration of some ideas. Now I think with new splitter
> it would be better to keep arglist unparsed. In this patch it was that
> if we have
> echo 1 2 3\;\$ ab$i;
> lexer returns (schematically)
> STRING(echo) STRING(1) STRING(2) STRING (3;$) STRING(abVAR(i)) SEMICOLON
> now I would do that it returns:
> STRING(echo 1 2 3\;\$ ab$i) SEMICOLON

Perhaps the best is to make a list of tokens we need to support.  With
such list we can determine how to detect them.

Thanks,
Marco

Re: Scripting support

[Vladimir Serbinenko]

Marco Gerards wrote:

>Vladimir Serbinenko <phcoder@gmail.com> writes:
>
>  
>
>
>I think we first have to determine:
>
>1) Which commands should be supported.
>  
>
What about 
&& || ! case do done elif else esac fi for function if in then until 
while { } [[ ]]
in scripting engine. I think that time is completely useless. What do 
you think about select?

>2) The smallest subset of commands that should be generated.
>
>  
>
I think
jmp
je/jne
invert (not for last return)
exec
regexp (check if string matches regexp)
push
pop
getelem (get element from list)

should be enough
External commands would be eval and [[
functions can be stored directly by parser

>
>Perhaps the best is to make a list of tokens we need to support.  With
>such list we can determine how to detect them.
>  
>
I think just comparing (after \ one character is not compared) might be 
enough.
; newline ;; && || ! case do done elif else esac fi for function if in 
then until while { } [[ ]]

>Thanks,
>Marco
>
>
>  
>
                                                             Vladimir 
'phcoder' Serbinenko

Re: Scripting support

[Marco Gerards]

Marco Gerards <metgerards@student.han.nl> writes:

> I think we first have to determine:
>
> 1) Which commands should be supported.
> 2) The smallest subset of commands that should be generated.
>
> Did you have a look at how bash does this?  Perhaps it even has a
> completely different approach...

I just had a look at bash.  It was a quick look, so I might be
mistaken, but I think it works like this:

First bash parses the script (in parse.y).  It uses a make_*_command
to generate stuff like if, for, case, etc (make_*command is in
make_cmd.c).  After that it executes the commands (in execute_cmd.c).

We could parse this script:

for i in 1 2 3
  do
    echo foo
    echo bar
    echo $i
  done
echo done!

into something like:

struct command
{
  command_type_t type;
};

struct for_command
{
  struct command command;
  
  list_t list;
  struct cmdlist_command *cmds;
}

/* A simple GRUB command.  */
struct simple_command
{
  struct command command;

  char *command;
};

/* Lines of commands.  */
struct cmdlist_command
{
  struct command command;

  struct command **cmdlist;
};

struct cmdlist_command script
 {
   grub_cmdtype_cmdlist,
   {
     { 
       cmd_type_for,
       {1, 2, 3},
       {
         grub_cmdtype_cmdlist,
         {
           { grub_cmdtype_simple, "echo foo" },
           { grub_cmdtype_simple, "echo bar" },
           { grub_cmdtype_simple, "echo $i" }
         }
       }
     },
     {
       grub_cmdtype_simple, " echo done!"
     }
 }


This is of course not the code what is generated.  And when
implementing it, you would use pointers.  But logically it is nested
that way.  After that you can execute it like this:

execute_cmd (struct command *cmd)
{
  switch (cmd->type)
   {
     case grub_cmdtype_if:
       {
         struct if_command *cmd_if = (struct if_command *) cmd;
         execute_if_cmd (cmd_if);
       }
     case grub_cmdtype_for:
      ...
   }
}


This is quite ugly code, but I hope you get the idea. :)

Vladimir, if you want I could implement the framework like I have in
mind.  It is not that much work.  The hard part is "filling in" the
framework so you get scripting support.  Please tell me what you
think, or we could talk about it on IRC.

Thanks,
Marco

Re: Scripting support

[Marco Gerards]

Vladimir Serbinenko <phcoder@gmail.com> writes:

>>I think we first have to determine:
>>
>>1) Which commands should be supported.
>>
>>
> What about && || ! case do done elif else esac fi for function if in
> then until while { } [[ ]]

This sounds fine to me.  Although I don't remember !, { } and [[ ]]
anymore.

> in scripting engine. I think that time is completely useless. What do
> you think about select?

I am not sure about select.

What is "time" you are talking about?  If you are talking about the
command time which measures the running time, it is just a command.
If we want it, we can easily implement it as a GRUB command.

>>2) The smallest subset of commands that should be generated.
>>
> I think

If we take the approach I described in my other email, we don't need
this.  So I wait for commenting about this.

>>Perhaps the best is to make a list of tokens we need to support.  With
>>such list we can determine how to detect them.
>>
>>
> I think just comparing (after \ one character is not compared) might
> be enough.

The \ is already handled by the state machine, when I commit it.  So
it is not something you have to worry about. :-)

> ; newline ;; && || ! case do done elif else esac fi for function if in
> then until while { } [[ ]]

Right.  And how about menu entries?  What was the last thing we agreed
about?

Re: Scripting support

[Yoshinori K. Okuji]

On Saturday 22 October 2005 04:46 pm, Vladimir Serbinenko wrote:
> I propose the following argorithm. Bison parser transforms the text-form
> to internal form of a linked list of commands with some special commands
> like jump if last value was true, ... . Think about assembler. Then it's
> executed by simple engine that works like CPU: it takes one command from
> a linked list. Executes it and changes the pointer so it points to the
> next command. A command is a structure containing:
>     variable showing it's type (normal command, jump, loop, end of loop,
> ...)
>     union of data necessary for necessary commands
>     pointer to next command.

Ahghh. Please avoid making yet another language. Since the language constructs 
in shell is so few, I don't see much benefit from such a translation.

Okuji