LFS Support for Sendfile

LFS Support for Sendfile

[Doug O'Neil]

Hello group.

First time poster. If this isn't the right place for this then please point
me in the right direction and accept my apology.

I'm using Linux 2.4.12 on a PIII

I've been playing with sendfile and noticed that it doesn't like 64 bit
filesystems ( _FILE_OFFSET_BITS=64). Is there a patch or kernel revision out
there that supports LFS for sendfile?  Or is there some method I can't see
to work around this.

I'm opening a descriptor on a disk partition. I don't need to support
individual block transfers larger than 2 Gig., but would like to serve up
partitions that are larger than 2Gig.

Thanks in advance

Doug

Re: LFS Support for Sendfile

[Doug McNaught]

"Doug O'Neil" <DougO@seven-systems.com> writes:

> Hello group.
> 
> First time poster. If this isn't the right place for this then please point
> me in the right direction and accept my apology.
> 
> I'm using Linux 2.4.12 on a PIII

That's pretty darn old.  Can you try it with a modern kernel (2.4.18)?

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.
   --T. J. Jackson, 1863

Re: LFS Support for Sendfile

[Matti Aarnio]

On Wed, Feb 27, 2002 at 08:22:10PM -0500, Doug McNaught wrote:
> "Doug O'Neil" <DougO@seven-systems.com> writes:
> > Hello group.
> > 
> > First time poster. If this isn't the right place for this then please point
> > me in the right direction and accept my apology.
> > 
> > I'm using Linux 2.4.12 on a PIII
> 
> That's pretty darn old.  Can you try it with a modern kernel (2.4.18)?

  Won't help.

The API (kernel syscall) as defined does not support LFS.

mm/filemap.c:
	asmlinkage ssize_t sys_sendfile(int out_fd, int in_fd, off_t *offset,
					 size_t count)

The problem there is that "off_t *" thing.  A loff_t in that spot would
solve the problem, but that needs rewriting the thing with a new syscall
number.   Trivial in itself, though.  (And changeing the count to 64-bit
value is equally trivial at the same time.)


There lurks another problem in the filesystems, though.

All SysV UFS tradition type filesystems with indirect indexing of
disk blocks will need more and more of metadata traversal to reach
actual data blocks on very large files.  It slows things down.

The "extent based" filesystems offer flatter performance, and while
I can't determine if ReiserFS is exactly of that type, it too offers
fast and flat performance.

> -Doug

/Matti Aarnio

Re: LFS Support for Sendfile

[Chris Wedgwood]

On Thu, Feb 28, 2002 at 10:03:25AM +0200, Matti Aarnio wrote:

    The API (kernel syscall) as defined does not support LFS.

I wonder does it really need to?  I mean, a loop calling sendfile for
2GB (or whatever) at a time is almost as good, if not better in some
ways.

    The "extent based" filesystems offer flatter performance, and
    while I can't determine if ReiserFS is exactly of that type, it
    too offers fast and flat performance.

Reiserfs (v3) isn't extent based but does perform pretty well.  When I
was messing large numbers of with (what at the time were) large files
of 50GB or so, XFS proved to be very effective.


  --cw

Re: LFS Support for Sendfile

[Hans Reiser]

Chris Wedgwood wrote:

>On Thu, Feb 28, 2002 at 10:03:25AM +0200, Matti Aarnio wrote:
>
>    The API (kernel syscall) as defined does not support LFS.
>
>I wonder does it really need to?  I mean, a loop calling sendfile for
>2GB (or whatever) at a time is almost as good, if not better in some
>ways.
>
>    The "extent based" filesystems offer flatter performance, and
>    while I can't determine if ReiserFS is exactly of that type, it
>    too offers fast and flat performance.
>
>Reiserfs (v3) isn't extent based but does perform pretty well.  When I
>was messing large numbers of with (what at the time were) large files
>of 50GB or so, XFS proved to be very effective.
>
>
>  --cw
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
extents are in reiserfs v4.0 (September ship date), and should offer 
much improved performance for large files.

Hans

Re: LFS Support for Sendfile

[Dan Maas]

>     [sendfile(2)] as defined does not support LFS.
>
> I wonder does it really need to?  I mean, a loop calling sendfile for
> 2GB (or whatever) at a time is almost as good, if not better in some
> ways.

The 'count' parameter is not the problem, it's the 'offset'. You can't send
any data beyond 2GB from the beginning of the file...

And in fact just two days ago I was in a situation where this would have
been desirable - I was sending parts of a captured DV video stream (the file
was 6GB)...

Regards,
Dan

Re: LFS Support for Sendfile

[Andrew Morton]

Dan Maas wrote:
> 
> >     [sendfile(2)] as defined does not support LFS.
> >
> > I wonder does it really need to?  I mean, a loop calling sendfile for
> > 2GB (or whatever) at a time is almost as good, if not better in some
> > ways.
> 
> The 'count' parameter is not the problem, it's the 'offset'. You can't send
> any data beyond 2GB from the beginning of the file...
> 
> And in fact just two days ago I was in a situation where this would have
> been desirable - I was sending parts of a captured DV video stream (the file
> was 6GB)...
> 

We need a sendfile64() system call.  It's really simple,  but
nobody did it.

-