[PATCH] Support for unknown size and not seekable files

[PATCH] Support for unknown size and not seekable files

[Szymon Janc]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

Hello,

First patch adds support for GRUB_FILE_SIZE_UNKNOW file size in grub_file_read 
and grub_file_seek functions. 

Second and third add GRUB_FILE_SIZE_UNKNOW support for gzio and xzio when 
dealing with not easily seekable files.


Should be usefull for reading compressed files over pxe. 

Testing and comments are welcome :)

-- 
Szymon K. Janc
szymon@janc.net.pl // GG: 1383435

[-- Attachment #2: 01_unknown_file_size_support.diff --]
[-- Type: text/x-patch, Size: 1917 bytes --]

=== modified file 'ChangeLog'
--- ChangeLog	2011-01-13 21:25:56 +0000
+++ ChangeLog	2011-01-14 23:53:37 +0000
@@ -1,3 +1,8 @@
+2011-01-15  Szymon Janc  <szymon@janc.net.pl>
+
+	* grub-core/kern/file.c (grub_file_read): Handle unknown file size.
+	(grub_file_seek): Likewise.
+
 2011-01-13  Vladimir Serbinenko  <phcoder@gmail.com>
 
 	* grub-core/fs/zfs/zfsinfo.c (grub_cmd_zfs_bootfs): Quote bootpath and

=== modified file 'grub-core/kern/file.c'
--- grub-core/kern/file.c	2010-09-05 11:05:36 +0000
+++ grub-core/kern/file.c	2011-01-14 23:53:37 +0000
@@ -132,15 +132,18 @@ grub_file_read (grub_file_t file, void *
 {
   grub_ssize_t res;
 
-  if (file->offset > file->size)
+  if (file->size != GRUB_FILE_SIZE_UNKNOWN)
     {
-      grub_error (GRUB_ERR_OUT_OF_RANGE,
-		  "attempt to read past the end of file");
-      return -1;
-    }
+      if (file->offset > file->size)
+	{
+	  grub_error (GRUB_ERR_OUT_OF_RANGE,
+		      "attempt to read past the end of file");
+	  return -1;
+	}
 
-  if (len == 0 || len > file->size - file->offset)
-    len = file->size - file->offset;
+      if (len == 0 || len > file->size - file->offset)
+	len = file->size - file->offset;
+    }
 
   /* Prevent an overflow.  */
   if ((grub_ssize_t) len < 0)
@@ -171,16 +174,22 @@ grub_file_close (grub_file_t file)
 grub_off_t
 grub_file_seek (grub_file_t file, grub_off_t offset)
 {
-  grub_off_t old;
+  grub_off_t old = file->offset;
+
+  if (file->size == GRUB_FILE_SIZE_UNKNOWN && offset > file->offset)
+    {
+      if (grub_file_read (file, NULL, offset - file->offset) < 0)
+	return -1;
+      else
+	return old;
+    }
 
   if (offset > file->size)
     {
-      grub_error (GRUB_ERR_OUT_OF_RANGE,
-		  "attempt to seek outside of the file");
+      grub_error (GRUB_ERR_OUT_OF_RANGE, "attempt to seek outside of the file");
       return -1;
     }
 
-  old = file->offset;
   file->offset = offset;
   return old;
 }


[-- Attachment #3: 02_notseekable_file_gzio.diff --]
[-- Type: text/x-patch, Size: 1875 bytes --]

=== modified file 'ChangeLog'
--- ChangeLog	2011-01-14 23:53:37 +0000
+++ ChangeLog	2011-01-15 00:03:32 +0000
@@ -1,5 +1,13 @@
 2011-01-15  Szymon Janc  <szymon@janc.net.pl>
 
+	* grub-core/io/gzio.c (test_header): For not seekable files: don't
+	check footer and set size to unknown.
+	(inflate_window): Set real size for unknown size file when it is
+	known.
+	(grub_gzio_read): Handle NULL buffer pointer.
+
+2011-01-15  Szymon Janc  <szymon@janc.net.pl>
+
 	* grub-core/kern/file.c (grub_file_read): Handle unknown file size.
 	(grub_file_seek): Likewise.
 

=== modified file 'grub-core/io/gzio.c'
--- grub-core/io/gzio.c	2010-12-26 20:15:31 +0000
+++ grub-core/io/gzio.c	2011-01-15 00:03:32 +0000
@@ -212,7 +212,7 @@ test_header (grub_file_t file)
 
   gzio->data_offset = grub_file_tell (gzio->file);
 
-  /* FIXME: don't do this on not easily seekable files.  */
+  if(grub_file_seekable(gzio->file))
   {
     grub_file_seek (gzio->file, grub_file_size (gzio->file) - 4);
     if (grub_file_read (gzio->file, &orig_len, 4) != 4)
@@ -224,6 +224,8 @@ test_header (grub_file_t file)
        But how can we know the real original size?  */
     file->size = grub_le_to_cpu32 (orig_len);
   }
+  else
+    file->size = GRUB_FILE_SIZE_UNKNOWN;
 
   initialize_tables (file);
 
@@ -1034,7 +1036,11 @@ inflate_window (grub_file_t file)
       if (! gzio->block_len)
 	{
 	  if (gzio->last_block)
-	    break;
+	    {
+	      if (file->size == GRUB_FILE_SIZE_UNKNOWN)
+		file->size = file->offset + gzio->wp;
+	      break;
+	    }
 
 	  get_new_block (file);
 	}
@@ -1186,9 +1192,11 @@ grub_gzio_read (grub_file_t file, char *
       if (size > len)
 	size = len;
 
-      grub_memmove (buf, srcaddr, size);
-
-      buf += size;
+      if (buf)
+	{
+	  grub_memmove (buf, srcaddr, size);
+	  buf += size;
+	}
       len -= size;
       ret += size;
       offset += size;


[-- Attachment #4: 03_notseekable_file_xzio.diff --]
[-- Type: text/x-patch, Size: 2201 bytes --]

=== modified file 'ChangeLog'
--- ChangeLog	2011-01-15 00:03:32 +0000
+++ ChangeLog	2011-01-15 00:08:24 +0000
@@ -1,5 +1,12 @@
 2011-01-15  Szymon Janc  <szymon@janc.net.pl>
 
+	* grub-core/io/xzio.c (test_footer): Return true for not seekable file.
+	(grub_xzio_open): FIXME comment removed.
+	(grub_xzio_read): Set real size for unknown size file when it is known.
+	Handle NULL buffer pointer.
+
+2011-01-15  Szymon Janc  <szymon@janc.net.pl>
+
 	* grub-core/io/gzio.c (test_header): For not seekable files: don't
 	check footer and set size to unknown.
 	(inflate_window): Set real size for unknown size file when it is

=== modified file 'grub-core/io/xzio.c'
--- grub-core/io/xzio.c	2010-12-26 20:15:31 +0000
+++ grub-core/io/xzio.c	2011-01-15 00:08:24 +0000
@@ -129,6 +129,10 @@ test_footer (grub_file_t file)
   grub_uint64_t uncompressed_size;
   grub_uint64_t records;
 
+  /* Don't test footer on not easily seekable files.  */
+  if (! grub_file_seekable(xzio->file))
+    return 1;
+
   grub_file_seek (xzio->file, xzio->file->size - FOOTER_MAGIC_SIZE);
   if (grub_file_read (xzio->file, footer, FOOTER_MAGIC_SIZE) !=
       FOOTER_MAGIC_SIZE
@@ -222,7 +226,6 @@ grub_xzio_open (grub_file_t io)
   xzio->buf.out_pos = 0;
   xzio->buf.out_size = XZBUFSIZ;
 
-  /* FIXME: don't test footer on not easily seekable files.  */
   if (!test_header (file) || !test_footer (file))
     {
       grub_errno = GRUB_ERR_NONE;
@@ -297,10 +300,14 @@ grub_xzio_read (grub_file_t file, char *
 	  /* Store first chunk of data in buffer.  */
 	  {
 	    grub_size_t delta = new_offset - (file->offset + ret);
-	    grub_memmove (buf, xzio->buf.out + (xzio->buf.out_pos - delta),
+
+	    if (buf)
+	      {
+		grub_memmove (buf, xzio->buf.out + (xzio->buf.out_pos - delta),
 			  delta);
+		buf += delta;
+	      }
 	    len -= delta;
-	    buf += delta;
 	    ret += delta;
 	  }
 	current_offset = new_offset;
@@ -308,7 +315,11 @@ grub_xzio_read (grub_file_t file, char *
       xzio->buf.out_pos = 0;
 
       if (xzret == XZ_STREAM_END)	/* Stream end, EOF.  */
-	break;
+	{
+	  if(file->size == GRUB_FILE_SIZE_UNKNOWN)
+	    file->size = file->offset + ret;
+	  break;
+	}
     }
 
   if (ret >= 0)

Re: [PATCH] Support for unknown size and not seekable files

[Seth Goldberg]

Hi,

   Thanks for this.  What's the goal of this patch?  It seems that it marks the 
size and unknown until it gets to the metadata that indicates the file's size, 
then updates the size to the correct size, right?  Where do you see this being 
useful?  It would seems that even with this patch, one would still need to 
read a gzipped file twice to be able to get its size, then allocate a 
contiguous buffer large enough to hold the entire thing (if that is desired 
(which, in some cases, it is :))).

  Thanks,
  --S

Quoting Szymon Janc, who wrote the following on Sat, 15 Jan 2011:

> Hello,
>
> First patch adds support for GRUB_FILE_SIZE_UNKNOW file size in grub_file_read
> and grub_file_seek functions.
>
> Second and third add GRUB_FILE_SIZE_UNKNOW support for gzio and xzio when
> dealing with not easily seekable files.
>
>
> Should be usefull for reading compressed files over pxe.
>
> Testing and comments are welcome :)
>
> -- 
> Szymon K. Janc
> szymon@janc.net.pl // GG: 1383435
>

Re: [PATCH] Support for unknown size and not seekable files

[Szymon Janc]

On Saturday 15 January 2011 01:37:13 Seth Goldberg wrote:
> Hi,
> 
>    Thanks for this.  What's the goal of this patch?  It seems that it marks
> the size and unknown until it gets to the metadata that indicates the
> file's size, then updates the size to the correct size, right?  Where do
> you see this being useful? 

The biggest problem now is that opening compressed files require seeking to the 
end to get uncompressed size so on not seekable fs this causes 
grub_file_open("foo") to read whole file. With this patchset this is no longer a 
case. So in that use case

file = grub_file_open ("foo");
while (grub_file_read (file, buf, sizeof (buf)) > 0) { ... }

foo is read only once.


> It would seems that even with this patch, one
> would still need to read a gzipped file twice to be able to get its size,
> then allocate a contiguous buffer large enough to hold the entire thing
> (if that is desired (which, in some cases, it is :))).

Wouldn't sth like that work?

char * ptr = 0;
char buf[0x10000];
unsigned size, size_total = 0;

grub_file_t file = grub_file_open ("foo");
while ( (size = grub_file_read (file, buf, sizeof (buf))) > 0)
{
    ptr = grub_realloc (ptr, size_total + size);
    grub_memcpy (ptr+size_total, buf, size);
    size_total += size;
}

-- 
Szymon K. Janc
szymon@janc.net.pl // GG: 1383435

Re: [PATCH] Support for unknown size and not seekable files

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

On 01/15/2011 01:15 PM, Szymon Janc wrote:
> On Saturday 15 January 2011 01:37:13 Seth Goldberg wrote:
>   
>> Hi,
>>
>>    Thanks for this.  What's the goal of this patch?  It seems that it marks
>> the size and unknown until it gets to the metadata that indicates the
>> file's size, then updates the size to the correct size, right?  Where do
>> you see this being useful? 
>>     
> The biggest problem now is that opening compressed files require seeking to the 
> end to get uncompressed size so on not seekable fs this causes 
> grub_file_open("foo") to read whole file. With this patchset this is no longer a 
> case. So in that use case
>
> file = grub_file_open ("foo");
> while (grub_file_read (file, buf, sizeof (buf)) > 0) { ... }
>
> foo is read only once.
>
>
>   
>> It would seems that even with this patch, one
>> would still need to read a gzipped file twice to be able to get its size,
>> then allocate a contiguous buffer large enough to hold the entire thing
>> (if that is desired (which, in some cases, it is :))).
>>     
> Wouldn't sth like that work?
>
>   
It's O(N^2).
You need to always at least double the allocation every time you exceed
already allocated buffer in order to get O(N)
> char * ptr = 0;
> char buf[0x10000];
> unsigned size, size_total = 0;
>
> grub_file_t file = grub_file_open ("foo");
> while ( (size = grub_file_read (file, buf, sizeof (buf))) > 0)
> {
>     ptr = grub_realloc (ptr, size_total + size);
>     grub_memcpy (ptr+size_total, buf, size);
>     size_total += size;
> }
>
>   


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]