gettext in scripts

gettext in scripts

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

Hello, all. It was requested to have a way to gettextize Super GRUB 
disk. Original request was to extend gettext with -s options but it may 
result in a mess to temporary variables. So I'd propose to rather use 
$"..." syntax with the exception that we don't expand it after 
translation to avoid some pitfalls. Sharing syntax with bash will allow 
to use normal xgettext --language=Shell.
The real problem is as to how to include SGD strings in grub.pot. I see 
following solutions:
1) Merge projects and put SGD inside GRUB2. It's possible if its authors 
want it but many more questions would need to be discussed.
2) Import sgd.pot into grub.pot regularly. This creates an 
administrative overhead and requires GRUB and scripts to have matching 
versions. I don't know if it's easy to merge .pot files
3) Just have a few strings for usage by external projects. This would 
heavily limit the extent of internationalisation and would result in a 
lot of feature requests one after another for this or that string.
4) Have some way to load .mo from another directory. Perhaps something 
like locale_dirs=$dir1:$dir2:$dir3. This creates some problems as to how 
and when to load files and how to store them. The easiest way consisting 
of having multiple opened gettext files and try each of it in turn. 
Trouble is that then when adding a single directory we need to either 
reload all files or compare the previous list with current to reload 
only changed directories.
Any other ideas?

-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko


[-- Attachment #2: gettext.diff --]
[-- Type: text/x-diff, Size: 3371 bytes --]

=== modified file 'grub-core/script/execute.c'
--- grub-core/script/execute.c	2012-02-04 11:21:21 +0000
+++ grub-core/script/execute.c	2012-02-10 22:24:15 +0000
@@ -26,6 +26,7 @@
 #include <grub/lib/arg.h>
 #include <grub/normal.h>
 #include <grub/extcmd.h>
+#include <grub/i18n.h>
 
 /* Max digits for a char is 3 (0xFF is 255), similarly for an int it
    is sizeof (int) * 3, and one extra for a possible -ve sign.  */
@@ -312,8 +313,8 @@
   struct grub_script_arg *arg = 0;
   struct grub_script_argv result = { 0, 0, 0 };
 
-  auto int append (char *s, int escape_type);
-  int append (char *s, int escape_type)
+  auto int append (const char *s, int escape_type);
+  int append (const char *s, int escape_type)
   {
     int r;
     char *p = 0;
@@ -379,12 +380,20 @@
 	      break;
 
 	    case GRUB_SCRIPT_ARG_TYPE_TEXT:
-	      if (grub_strlen (arg->str) &&
+	      if (arg->str[0] &&
 		  grub_script_argv_append (&result, arg->str,
 					   grub_strlen (arg->str)))
 		goto fail;
 	      break;
 
+	    case GRUB_SCRIPT_ARG_TYPE_GETTEXT:
+	      {
+		const char *t = _(arg->str);
+		if (grub_script_argv_append (&result, t, grub_strlen (t)))
+		  goto fail;
+	      }
+	      break;
+
 	    case GRUB_SCRIPT_ARG_TYPE_DQSTR:
 	    case GRUB_SCRIPT_ARG_TYPE_SQSTR:
 	      if (append (arg->str, 1))

=== modified file 'grub-core/script/yylex.l'
--- grub-core/script/yylex.l	2012-02-03 10:56:49 +0000
+++ grub-core/script/yylex.l	2012-02-10 22:20:27 +0000
@@ -131,15 +131,17 @@
 SQCHR           [^\']
 DQCHR           {ESC}|[^\\\"]
 DQSTR           \"{DQCHR}*\"
+I18NSTR         \$\"{DQCHR}*\"
 SQSTR           \'{SQCHR}*\'
 SPECIAL         \?|\#|\*|\@
 VARIABLE        ${NAME}|$\{{NAME}\}|${DIGITS}|$\{{DIGITS}\}|${SPECIAL}|$\{{SPECIAL}\}
-WORD            ({CHAR}|{DQSTR}|{SQSTR}|{ESC}|{VARIABLE})+
+WORD            ({CHAR}|{DQSTR}|{SQSTR}|{ESC}|{VARIABLE}|{I18NSTR})+
 
 MULTILINE       {WORD}?((\"{DQCHR}*)|(\'{SQCHR}*)|(\\\n))
 
 %x              SPLIT
 %x              DQUOTE
+%x              I18NQUOTE
 %x              SQUOTE
 %x              VAR
 
@@ -215,6 +217,10 @@
                   yy_push_state (SQUOTE, yyscanner);
                   ARG (GRUB_SCRIPT_ARG_TYPE_TEXT);
                 }
+  "\$\""        {
+                  yy_push_state (I18NQUOTE, yyscanner);
+                  ARG (GRUB_SCRIPT_ARG_TYPE_GETTEXT);
+                }
   \$            {
                   yy_push_state (VAR, yyscanner);
                   ARG (GRUB_SCRIPT_ARG_TYPE_TEXT);
@@ -280,6 +286,18 @@
   (.|\n)        { COPY (yytext, yyleng); }
 }
 
+<I18NQUOTE>{
+  \\\\          { COPY ("\\", 1); }
+  \\\"          { COPY ("\"", 1); }
+  \\\n          { /* ignore */ }
+  [^\"\\\n]+    { COPY (yytext, yyleng); }
+  \"            {
+                  yy_pop_state (yyscanner);
+                  ARG (GRUB_SCRIPT_ARG_TYPE_GETTEXT);
+                }
+  (.|\n)        { COPY (yytext, yyleng); }
+}
+
 <<EOF>>         {
                   yypop_buffer_state (yyscanner);
 		  yyextra->lexerstate->eof = 1;

=== modified file 'include/grub/script_sh.h'
--- include/grub/script_sh.h	2011-11-11 19:34:37 +0000
+++ include/grub/script_sh.h	2012-02-10 20:30:33 +0000
@@ -53,6 +53,7 @@
 {
   GRUB_SCRIPT_ARG_TYPE_VAR,
   GRUB_SCRIPT_ARG_TYPE_TEXT,
+  GRUB_SCRIPT_ARG_TYPE_GETTEXT,
   GRUB_SCRIPT_ARG_TYPE_DQVAR,
   GRUB_SCRIPT_ARG_TYPE_DQSTR,
   GRUB_SCRIPT_ARG_TYPE_SQSTR,

Re: gettext in scripts

[Andreas Born]

Hi,

this is a great feature! Our LiveCD project (Salix Live) uses grub as 
bootloader and we offer a localized boot menu, which is - at least last 
time I checked - roughly implemented like SGD's localization. So not 
only SGD could benefit from such an infrastructure and this is something 
we've been looking for since the very beginning.

Am 10.02.2012 23:54, schrieb Vladimir 'φ-coder/phcoder' Serbinenko:
> Hello, all. It was requested to have a way to gettextize Super GRUB 
> disk. Original request was to extend gettext with -s options but it 
> may result in a mess to temporary variables. So I'd propose to rather 
> use $"..." syntax with the exception that we don't expand it after 
> translation to avoid some pitfalls. Sharing syntax with bash will 
> allow to use normal xgettext --language=Shell.
> The real problem is as to how to include SGD strings in grub.pot. I 
> see following solutions:
> 1) Merge projects and put SGD inside GRUB2. It's possible if its 
> authors want it but many more questions would need to be discussed.
> 2) Import sgd.pot into grub.pot regularly. This creates an 
> administrative overhead and requires GRUB and scripts to have matching 
> versions. I don't know if it's easy to merge .pot files
> 3) Just have a few strings for usage by external projects. This would 
> heavily limit the extent of internationalisation and would result in a 
> lot of feature requests one after another for this or that string.
Those solutions would only allow SGD to offer localization. Except maybe 
3, but I'd say that one is of very limited use and irrealistic 
especially regarding the maintenance for grub.
> 4) Have some way to load .mo from another directory. Perhaps something 
> like locale_dirs=$dir1:$dir2:$dir3. This creates some problems as to 
> how and when to load files and how to store them. The easiest way 
> consisting of having multiple opened gettext files and try each of it 
> in turn. Trouble is that then when adding a single directory we need 
> to either reload all files or compare the previous list with current 
> to reload only changed directories.
That sounds like a much better idea as other projects could also use 
localization without increasing the workload for grub. Actually this 
looks like my favorite now. What do you think would be the 
perfomance/speed penalty regarding loading of the directories and 
accessing a translation?

I think for common use cases it would be enough to add additionally to 
the already implemented locale_dir=$dir1 a second environment variable 
like additional_locale_dir=$dir2. That would overcome the problem of 
reloading all directories/ comparing lists. It offers less flexibility, 
but a too long list of files to search would probably have a negative 
performance penalty anyway.

> Any other ideas?
Similarly to 2, but the other way round:
5) Import grub.pot to sgd.pot (or whatever other project). This is 
probably the easiest solution for grub with no additional workload. 
Regarding projects building on grub this is not a perfect solution and 
makes it harder to use such functionality. But it would be already a 
great step forward and the maintenance burden for other projects is by 
far not irrealistic. Given an easy-to-use, suitable and efficient tool 
(maybe already existent?) this solution is more than acceptable.

Still solution 4 seems way cleaner to me as it separates the strings of 
grub and other projects, which I imagine could be better in the one or 
other situation.
Specifically there could be problems when a user runs grub-install or 
equivalent and thus overwrites the .mo files with a different version. 
Or multiple projects each with its own translation could use the same 
grub installation.

Andreas

Re: gettext in scripts

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On 11.02.2012 01:21, Andreas Born wrote:
> I think for common use cases it would be enough to add additionally to
> the already implemented locale_dir=$dir1 a second environment variable
> like additional_locale_dir=$dir2. That would overcome the problem of
> reloading all directories/ comparing lists. It offers less
> flexibility, but a too long list of files to search would probably
> have a negative performance penalty anyway. 
I think everyone reached this consensus. Additionally it makes it
similar to bash with TEXTDOMAIN variable. So the use is
secondary_locale_dir=<my .mo directory>
echo $"Text to translate"

Because I discovered an important bug in gettext (inability to disable
it or change locale_dir) it needed a restructure and squeezing this was
trivial. Given gettext importance I did so.

However currently there is no "printf" in grub and it would be too big
for freeze exception. Lack of printf puts limits to localisation.

-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]