a compression tool useful with constrained memory

a compression tool useful with constrained memory

[u-vpoa]

Hello all,

I'd like to contribute a tool from the last millenium which worked fine
among others on 16-bit Venix/86. It is now BSD licensed, any and all
related patents expired since then.

It compiles fine by bcc, works faster and produces much better compression
than the 12-bit compress. Tested under elksemu.

The source is 22K bzip2 compressed, 140K uncompressed
(37K if compressed by itself :)
What would be a suitable way to submit it?

Regards,
Rl

Re: a compression tool useful with constrained memory

[Jody Bruchon]

On 4/26/2015 8:29 AM, u-vpoa@aetey.se wrote:
> What would be a suitable way to submit it?
Send it to me and I'll integrate it.

Re: a compression tool useful with constrained memory

[u-vpoa]

On Sun, Apr 26, 2015 at 08:36:19AM -0400, Jody Bruchon wrote:
> On 4/26/2015 8:29 AM, u-vpoa@aetey.se wrote:
> >What would be a suitable way to submit it?
> Send it to me and I'll integrate it.

Done!

Rl

Status updates: compression tool, kernel fixes, BCC features, etc.

[Jody Bruchon]

On 4/26/2015 9:11 AM, u-vpoa@aetey.se wrote:
> Done!
'prems' is now integrated in the ELKS elkscmd tree. I haven't tested it 
yet; I've only stuck it in there and made it compile, so feel free to 
try it out and let me know if any additional changes are needed. It is 
included by default in 'full3' images and the binary name is /bin/pres.

On 4/11/2015 3:33 PM, MFLD wrote:
 > This patch tries to solve the problem described in the "Question on 
import symbols generated by BCC

I committed your other BCC patches to my dev86 tree, but this particular 
patch causes libc compilation to fail with "already defined" import 
errors. If you figure out how to fix that and re-submit, I'll test again 
and commit.

On 4/11/2015 1:43 PM, Juan Perez-Sanchez wrote:
 > The attached patch does: (lots of good things)

Committed!

On 4/1/2015 7:22 AM, Nils Stec wrote:
 > Here's my updated patch. I think this attempt is better than the 
first, if you still find something annoying i will fix/change it.

I will need to rewrite the code in the patch to match my current repo 
version before I can commit. It'll take more time than I have at the 
immediate moment. I've got it in my work queue and will let everyone 
know when I've added it.

On 4/12/2015 11:43 AM, MFLD wrote:
 > I send you the HTML file directly, because the mailer daemon refuses 
to forward it (as HTML).

Thanks! I'll upload this when I am logged into SourceForge's systems again.

-Jody

Re: a compression tool useful with constrained memory

[u-vpoa]

[-- Attachment #1: Type: text/plain, Size: 77 bytes --]

Hello,

Here comes a patch (removing some unnecessary size limitations).

Rl

[-- Attachment #2: prems.patch20150426-20150427 --]
[-- Type: text/plain, Size: 7118 bytes --]

--- PRES/pres.c.ori	2015-04-27 22:04:01.705132713 +0200
+++ PRES/pres.c	2015-04-27 22:11:45.855097441 +0200
@@ -3,6 +3,14 @@
  *
  * A library (prem/malprem) and a cli frontend to it (pres)
  *
+ * version 0.2015-04-27
+ *         removed undue data stream size limitations,
+ *         now it should work for up to 2GB files,
+ *         bigger streams will be compressed alright but
+ *         the diagnostic messages could become wrong after 2GB
+ * version 0.2015-04-26
+ *         initial public release
+ *
  * Usage: pres [-dcfFqii] [file ...]
  * Flags:
  *      -d:         Decompress instead of compressing
@@ -246,8 +254,8 @@
     exit(exit_stat);
 }
 
-long int enk; /* input data length */
-long int elk; /* compressed length */
+unsigned long int enk; /* input data length */
+unsigned long int elk; /* compressed length */
 
 /*
  * Compressing input to output (one file)
@@ -267,9 +275,10 @@
         exit( 1 );
     }
     enk = elk = 0;
-    for( ;; ){
+    b_n = 0;
+    for( ;; ++b_n ){
+ /* in output we roll over every 2^16 blocks regardless the b_n type size */
         d = enb;
-        b_n = enk / BLON; /* the future block number */
         /* fill the output buffer */
         while( d - enb < BLON && (c=getchar()) != EOF ){
             ++enk;
@@ -311,10 +320,10 @@
     /*
      * tell the result
      */
-    if(wcat_flg == 0 && !quiet) {
-        fprintf( stderr, "Compression: %d%%",
-                enk > 0 ?
-                ( (100 * ( enk - elk )) / enk ) :
+    if(wcat_flg == 0 && !quiet) { long dif = (long)enk-(long)elk;
+        fprintf( stderr, "Compression: %d%%",
+                enk ?
+                (int)((dif>21474836)?(dif/((long)enk/100)):(100*dif/(long)enk)) :
                 0 );
     }
     if(elk > enk)    /* exit(2) if no compression */
@@ -439,12 +448,13 @@
                 continue;
             }
         }
-        oblk = it_blk + 1;
+        oblk = (it_blk + 1)&65535; /* ensure rolling over every 2^16 blocks */
         lon = it_lon;
         if( lon == BLON ){ /* not the last block */
-            if( (i=serch()) == EOF    /* end ??? */
-             || it_blk != oblk )  /* out of order */
-                    continue; /* do not trust this block */
+            if( ((i=serch()) == EOF    /* end ??? */
+                 || it_blk != oblk)    /* out of order */
+                 && !ignore )
+                    continue; /* do not trust the freshly uncompressed block */
         } else { /* looks like the last one */
             if( ignore && (i=serch()) != EOF ){
                 /* not last, look further! */
@@ -457,8 +467,8 @@
                     continue; /* ignoring the "last" block indication */
             }
         }
-        if( (j=oblk-b_blk-1) != 0 ){
-            /* the next block number in order, but here there was a skip */
+        if( (j=oblk-((b_blk+1)&65535)) != 0 ){
+            /* there was a skip */
             if( fseek( stdout, (long)oblk*(long)BLON, 0 ) < 0 )
                 /* can not seek (pipe) */
                 if( j > 0 && j <= 10 ) /* skipped blocks
--- PRES/flparc.c.ori	2015-04-27 22:04:06.568430732 +0200
+++ PRES/flparc.c	2015-04-27 22:14:26.303900666 +0200
@@ -2,6 +2,14 @@
  * FLPARC - recording/reading data on removable data, with compression
  * (c) RL 1990-2015
  *
+ * version 0.2015-04-27
+ *         removed undue data stream size limitations,
+ *         now it should work for up to 2GB,
+ *         bigger streams will be compressed alright but
+ *         the diagnostic messages would become wrong after 2GB
+ * version 0.2015-04-26
+ *         initial public release
+ *
  * $Header: flparc.c,v 1.2 91/01/24 14:11:35 pro Exp $
  * $Log:	flparc.c,v $
  * Revision 1.2  91/01/24  14:11:35  pro
@@ -36,7 +44,7 @@
 int bi = 0;       /* Index in the blocking buffer */
 unsigned int bimaks; /* Size of the blocking buffer */
 long skp = 0;     /* Pointer in the archive file */
-long num = 0;     /* Number of the bytes input for compression */
+unsigned long num = 0;     /* Number of the bytes input for compression */
 int  flp = -1;    /* Number of the filled media */
 int verb = 0;     /* Tell the current compression ratio */
 int ignore = 0;   /* Ignore read errors at decompression */
@@ -295,11 +303,14 @@
 	    " Skriberaro! Bloko %d\n"), skp/B_LON );
 	    exit_stat = 5;
     }
-    if( num > 0 && verb ) fprintf( stderr, ediag(
-	" Compressed: %d%%\n",
-	" Kunpremite: %d%%\n"),
-	(100*(num-((long)flp*dlon*B_LON+skp+bi)))/num );
-/* note this works up to ca 21MB of size reduction (overflow of 100*(long)) */
+    if( num > 0 && verb ){ long dif = (long)num-((long)flp*dlon*B_LON+skp+bi);
+        fprintf( stderr, ediag(
+	    " Compressed: %d%%\n",
+	    " Kunpremite: %d%%\n"),
+	    num ?
+            (int)((dif>21474836)?(dif/((long)num/100)):(100*dif/(long)num)) :
+            0 );
+    }
     skp = 0;
     bi = 0;
 }
@@ -310,12 +321,13 @@
     char c, d;
 
     ++flp;
-    if( premado && verb && flp > 0 ){
+    if( premado && verb && flp > 0 ){ long dif = (long)num-(long)flp*dlon*B_LON;
 	fprintf( stderr, ediag(
-	    " Currently compressed %d%%\n",
-	    " Nuntempa premado %d%%\n"),
- (100*(num-((long)flp*dlon*B_LON)))/num );
-/* note this works up to ca 21MB of size reduction (overflow of 100*(long)) */
+	    " Currently compressed %d%%\n",
+	    " Nuntempa premado %d%%\n"),
+	    num ?
+            (int)((dif>21474836)?(dif/((long)num/100)):(100*dif/(long)num)) :
+            0 );
     }
     fprintf( stderr, ediag(
 	"\
@@ -375,9 +387,10 @@
         exit_stat = 5;
         return;
     }
-    for( ;; ){
+    b_n = 0;
+    for( ;; ++b_n ){
+ /* in output we roll over every 2^16 blocks regardless the b_n type size */
         d = enb;
-        b_n = num / BLON; /* the future block number */
         /* fill the input buffer */
         while( d - enb < BLON && (c=getchar()) != EOF ){
             ++num;
@@ -541,12 +554,13 @@
                 continue;
             }
         }
-        oblk = it_blk + 1;
+        oblk = (it_blk + 1)&65535; /* ensure rolling over every 2^16 blocks */
         lon = it_lon;
         if( lon == BLON ){ /* not the last block */
-            if( (i=serch()) == EOF    /* error */
-             || it_blk != oblk )  /* out of order */
-                    continue; /* don't trust this block */
+            if( ((i=serch()) == EOF    /* error */
+                 || it_blk != oblk )   /* out of order */
+                 && !ignore )
+                    continue; /* do not trust the freshly uncompressed block */
         } else { /* looks like the last block */
             if( ignore ){
                 /* not the last, we may need to look further! */
@@ -566,8 +580,8 @@
                 }
             }
         }
-        if( (j=oblk-b_blk-1) != 0 ){
-            /* the next block number in order, but here there was a skip */
+        if( (j=oblk-((b_blk+1)&65535)) != 0 ){
+            /* there was a skip */
             if( fseek( stdout, (long)oblk*(long)BLON, 0 ) < 0 )
                 /* can not seek (pipe) */
                 if( j > 0 && j <= 10 ) /* skipped blocks

another fix for huge data (Re: a compression tool useful with constrained memory)

[u-vpoa]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

Noticed a situation where we would divide by zero:

+ * version 0.2015-04-29
+ *         avoid division by zero when the input length
+ *         differs from N*4GB less than 100 bytes

Attaching a patch which should apply cleanly (build tested with
a different patch version, for the original source format).

Regards,
Rl

[-- Attachment #2: patch.20150429.bz2 --]
[-- Type: application/octet-stream, Size: 1261 bytes --]