Contiguous file sequences

Contiguous file sequences

[Daire Byrne]

Hi,

I have been trying to figure out how to lay down a file sequence (e.g.
images) such that they are guaranteed to always be contiguous on disk
(i.e. no block gaps between them). Currently if I write a sequence to
disk things like "filestreams" help keep everything in the same AG and
the allocation algorithm seems to prefer to try and place files next
to eachother but without the filesystem knowing the total size of the
sequence there are always likely to be gaps in the blocks where
existing data has been written. So even if the first file is written
completely contiguously to disk there is no way to guarantee that
there is contiguous free space after it to write the rest of the
images.

What I really want is to be able to find and reserve enough space for
the entire sequence and then write the files into that big contiguous
range. I tried to do this with xfs_io hoping that the allocator would
just know what I wanted and do the right thing (ever the optimist...).
So something like this:

  # find and reserve a big chunk to fit all my files in
  xfs_io -f -c "resvsp 0 136314880" -c "bmap -v" $DIR/test.0

  # now shrink it keeping the start block
  xfs_io -f -c "freesp 13631488 0" -c "bmap -v" $DIR/test.0

  # now write a bunch of files and hope they continue from test.0 on disk
  dd if=/dev/zero of=$DIR/test.0 bs=1M count=13 conv=nocreat,notrunc
  for  x in `seq 1 4`; do
      dd if=/dev/zero of=$DIR/test.$x bs=1M count=13 conv=notrunc
  done

But a new allocation is made for the first new file in the sequence
elsewhere on disk and I don't know how to get it to use the large
chunk of free contiguous space after the "test.0" file instead.
Another option might be to create a single contiguous large file,
concatenate all the images into it and then split it up on disk using
offsets but I don't think such a thing is even possible? I always know
the image sequence size beforehand, all images are exactly the same
size and I can control/freeze the filesystem access if needed.

Anybody got any suggestions? It *seems* like something that should be
possible and would be useful.

Daire

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Eric Sandeen]

Daire Byrne wrote:
> Hi,
> 
> I have been trying to figure out how to lay down a file sequence (e.g.
> images) such that they are guaranteed to always be contiguous on disk
> (i.e. no block gaps between them). 

There's no mechanism to guarantee that.

Why is this the goal, what are you trying to achieve?

> Currently if I write a sequence to
> disk things like "filestreams" help keep everything in the same AG and
> the allocation algorithm seems to prefer to try and place files next
> to eachother but without the filesystem knowing the total size of the
> sequence there are always likely to be gaps in the blocks where
> existing data has been written. 

preallocation of each image before writing would help make it more
likely that each image is itself contiguous (but again this is not
-guaranteed-)

> So even if the first file is written
> completely contiguously to disk there is no way to guarantee that
> there is contiguous free space after it to write the rest of the
> images.
> 
> What I really want is to be able to find and reserve enough space for
> the entire sequence and then write the files into that big contiguous
> range. I tried to do this with xfs_io hoping that the allocator would
> just know what I wanted and do the right thing (ever the optimist...).

:)

> So something like this:
> 
>   # find and reserve a big chunk to fit all my files in
>   xfs_io -f -c "resvsp 0 136314880" -c "bmap -v" $DIR/test.0
> 
>   # now shrink it keeping the start block
>   xfs_io -f -c "freesp 13631488 0" -c "bmap -v" $DIR/test.0
> 
>   # now write a bunch of files and hope they continue from test.0 on disk
>   dd if=/dev/zero of=$DIR/test.0 bs=1M count=13 conv=nocreat,notrunc
>   for  x in `seq 1 4`; do
>       dd if=/dev/zero of=$DIR/test.$x bs=1M count=13 conv=notrunc
>   done
> 
> But a new allocation is made for the first new file in the sequence
> elsewhere on disk and I don't know how to get it to use the large
> chunk of free contiguous space after the "test.0" file instead.

You can't specify a starting block for any given file I'm afraid.

> Another option might be to create a single contiguous large file,
> concatenate all the images into it and then split it up on disk using
> offsets but I don't think such a thing is even possible? I always know
> the image sequence size beforehand, all images are exactly the same
> size and I can control/freeze the filesystem access if needed.
> 
> Anybody got any suggestions? It *seems* like something that should be
> possible and would be useful.

This would be pretty low-level control of the allocator by userspace.

I'll just go back and ask what problem you're trying to solve?  There
may be a better (i.e. currently existing) solution.

-Eric

> Daire
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Matthias Schniedermeyer]

On 22.09.2010 12:01, Daire Byrne wrote:
> Hi,

....


There is actually a VERY easy solution nowadays. SSD(s)

It doesn't matter if files are continuous or not, SSDs don't care (at 
least the better ones)

If you care about "worst case" write performance, zap the whole thing 
before usage (a.k.a.: trim) and write in chunks that are sized and 
aligned to erase-blocks (or just write in big chunks like 1MB or more). 
That should be more or less be enough to prevent latency spikes, that 
CAN plague SSDs.

What exactly you need obviously depends on the usual factors:
Needed sustained read/write bandwith, size, money, etc. etc.





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Dave Chinner]

On Wed, Sep 22, 2010 at 10:50:52PM +0200, Matthias Schniedermeyer wrote:
> On 22.09.2010 12:01, Daire Byrne wrote:
> > Hi,
> 
> ....
> 
> 
> There is actually a VERY easy solution nowadays. SSD(s)
> 
> It doesn't matter if files are continuous or not, SSDs don't care (at 
> least the better ones)

IO size and alignment still matters for performance on SSDs, just
not as much as for spinning rust.  That is, large sqeuental IOs are
still much faster on SSDs than lots of small random IOs.  Even on a
fusionIO card, large IOs will get 2-3x the _best case_ throughput of
small, random IOs.

> If you care about "worst case" write performance, zap the whole thing 
> before usage (a.k.a.: trim) and write in chunks that are sized and 
> aligned to erase-blocks (or just write in big chunks like 1MB or more). 
> That should be more or less be enough to prevent latency spikes, that 
> CAN plague SSDs.

Which is exactly why filesystem layout still matters with SSDs. If
the filesystem lays out files contiguously, and does IO mostly in
large chunks, then the SSDs age a lot better (as does the
filesystem) and performance shouldn't degrade significntly with
time....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Daire Byrne]

Eric,

On Wed, Sep 22, 2010 at 9:10 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> Daire Byrne wrote:
>> Hi,
>>
>> I have been trying to figure out how to lay down a file sequence (e.g.
>> images) such that they are guaranteed to always be contiguous on disk
>> (i.e. no block gaps between them).
>
> There's no mechanism to guarantee that.
>
> Why is this the goal, what are you trying to achieve?

I am essentially trying to play back a large frame sequence and trying
to minimise seeks as it can lead to sporadic slowdowns on a SATA based
RAID.

>> Currently if I write a sequence to
>> disk things like "filestreams" help keep everything in the same AG and
>> the allocation algorithm seems to prefer to try and place files next
>> to eachother but without the filesystem knowing the total size of the
>> sequence there are always likely to be gaps in the blocks where
>> existing data has been written.
>
> preallocation of each image before writing would help make it more
> likely that each image is itself contiguous (but again this is not
> -guaranteed-)
>
>> So even if the first file is written
>> completely contiguously to disk there is no way to guarantee that
>> there is contiguous free space after it to write the rest of the
>> images.
>>
>> What I really want is to be able to find and reserve enough space for
>> the entire sequence and then write the files into that big contiguous
>> range. I tried to do this with xfs_io hoping that the allocator would
>> just know what I wanted and do the right thing (ever the optimist...).
>
> :)
>
>> So something like this:
>>
>>   # find and reserve a big chunk to fit all my files in
>>   xfs_io -f -c "resvsp 0 136314880" -c "bmap -v" $DIR/test.0
>>
>>   # now shrink it keeping the start block
>>   xfs_io -f -c "freesp 13631488 0" -c "bmap -v" $DIR/test.0
>>
>>   # now write a bunch of files and hope they continue from test.0 on disk
>>   dd if=/dev/zero of=$DIR/test.0 bs=1M count=13 conv=nocreat,notrunc
>>   for  x in `seq 1 4`; do
>>       dd if=/dev/zero of=$DIR/test.$x bs=1M count=13 conv=notrunc
>>   done
>>
>> But a new allocation is made for the first new file in the sequence
>> elsewhere on disk and I don't know how to get it to use the large
>> chunk of free contiguous space after the "test.0" file instead.
>
> You can't specify a starting block for any given file I'm afraid.

Somebody pointed me at this which looks fairly promising:

  http://oss.sgi.com/archives/xfs/2006-07/msg01005.html

I'm still trying to get my head around how I would actually write a
userspace app/script to use it but I think it should allow me to do
what I want. It would be good if I could script it through xfs_io. I'd
really like a script where I could point it at a directory and it
would do something like:

  1. count total space used by file sequence
  2. find start block for that much contiguous space on disk (or as
much of it as possible)
  3. allocate the files using the start block one after another on disk

>> Another option might be to create a single contiguous large file,
>> concatenate all the images into it and then split it up on disk using
>> offsets but I don't think such a thing is even possible? I always know
>> the image sequence size beforehand, all images are exactly the same
>> size and I can control/freeze the filesystem access if needed.
>>
>> Anybody got any suggestions? It *seems* like something that should be
>> possible and would be useful.
>
> This would be pretty low-level control of the allocator by userspace.
>
> I'll just go back and ask what problem you're trying to solve?  There
> may be a better (i.e. currently existing) solution.

The "realtime" option is sometimes suggested as a way to do sequence
streaming but I'd really rather avoid that. It seems to me like the
option to allocate a sequence of files end on end in a known chunk of
contiguous space is something that would be useful in the normal
operating mode. SSDs are an option but they ain't cheap for the amount
of storage I require and besides I know that when the sequence is
written contiguously on disk my current setup can reach the required
speeds.

Thanks,

Daire

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 877 bytes --]

On Mittwoch, 22. September 2010 Eric Sandeen wrote:
> preallocation of each image before writing would help make it more
> likely that each image is itself contiguous (but again this is not
> -guaranteed-)
 
If all images were 1-2MB large, and if there is no parallel create, but 
all images are copied after each other, wouldn't the mount option 
"allocsize=2M" be sufficient to ensure pics are contigous? I guess XFS 
would place them behind each other on disk, or why should there be gaps 
between?

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Eric Sandeen]

Daire Byrne wrote:
> Eric,
> 
> On Wed, Sep 22, 2010 at 9:10 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>> Daire Byrne wrote:
>>> Hi,
>>>
>>> I have been trying to figure out how to lay down a file sequence (e.g.
>>> images) such that they are guaranteed to always be contiguous on disk
>>> (i.e. no block gaps between them).
>> There's no mechanism to guarantee that.
>>
>> Why is this the goal, what are you trying to achieve?
> 
> I am essentially trying to play back a large frame sequence and trying
> to minimise seeks as it can lead to sporadic slowdowns on a SATA based
> RAID.

Ok - and you've really seen allocation patterns that cause the playback
to slow down?  xfs_bmap information for a few sequential files that were
this far off would be interesting to see.

Are you certain that it's seekiness causing the problem?  A great way
to visualize it would be to use the seekwatcher application while you
run a problematic file sequence.

...

>> You can't specify a starting block for any given file I'm afraid.
> 
> Somebody pointed me at this which looks fairly promising:
> 
>   http://oss.sgi.com/archives/xfs/2006-07/msg01005.html

Yeah, that never got merged, but I think it still could be.

It's only half your battle though, you need to find that contiguous
space first, then specify the start block for it with the interface
above.

> I'm still trying to get my head around how I would actually write a
> userspace app/script to use it but I think it should allow me to do
> what I want. It would be good if I could script it through xfs_io. I'd
> really like a script where I could point it at a directory and it
> would do something like:
> 
>   1. count total space used by file sequence
>   2. find start block for that much contiguous space on disk (or as
> much of it as possible)
>   3. allocate the files using the start block one after another on disk
> 
>>> Another option might be to create a single contiguous large file,
>>> concatenate all the images into it and then split it up on disk using
>>> offsets but I don't think such a thing is even possible? I always know
>>> the image sequence size beforehand, all images are exactly the same
>>> size and I can control/freeze the filesystem access if needed.
>>>
>>> Anybody got any suggestions? It *seems* like something that should be
>>> possible and would be useful.
>> This would be pretty low-level control of the allocator by userspace.
>>
>> I'll just go back and ask what problem you're trying to solve?  There
>> may be a better (i.e. currently existing) solution.
> 
> The "realtime" option is sometimes suggested as a way to do sequence
> streaming but I'd really rather avoid that. It seems to me like the
> option to allocate a sequence of files end on end in a known chunk of
> contiguous space is something that would be useful in the normal
> operating mode.

It would be, but it's not there now.  Also, without some more complexity
it'd still probably end up being a best effort rather than a guarantee,
but some hints from userspace might be better than nothing.

-ERic

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Daire Byrne]

Eric,

On Mon, Sep 27, 2010 at 5:08 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> Daire Byrne wrote:
>>> Why is this the goal, what are you trying to achieve?
>>
>> I am essentially trying to play back a large frame sequence and trying
>> to minimise seeks as it can lead to sporadic slowdowns on a SATA based
>> RAID.
>
> Ok - and you've really seen allocation patterns that cause the playback
> to slow down?  xfs_bmap information for a few sequential files that were
> this far off would be interesting to see.
>
> Are you certain that it's seekiness causing the problem?  A great way
> to visualize it would be to use the seekwatcher application while you
> run a problematic file sequence.

I'm certain that the seekiness is the culprit. The image files are
pretty big and require 400MB/s+ speeds to play them back at full rate.
I can play a sequence which is aligned perfectly on disk just fine
(readahead) but when seeks are required between frames the framerate
drops noticeably. I'm using SATA disks which probably doesn't help
matters.

>>> You can't specify a starting block for any given file I'm afraid.
>>
>> Somebody pointed me at this which looks fairly promising:
>>
>>   http://oss.sgi.com/archives/xfs/2006-07/msg01005.html
>
> Yeah, that never got merged, but I think it still could be.
>
> It's only half your battle though, you need to find that contiguous
> space first, then specify the start block for it with the interface
> above.

I played around with the patch and I think I have a way to do what I
want using something like:

# allocate a big file that all the frames can fit into and hope it is contiguous
BLOCK=`xfs_io -f -c "resvsp 0 $TOTALSIZE" -c "freesp $FRAMESIZE 0" -c
"pwrite 0 1" -c "bmap" $DIR/test.0 | grep "0: \[" | sed 's/\../ /g' |
cut -f5 -d" "`
for x in `seq 1 $FRAMES`; do
    allocnear $DIR/test.$x $BLOCK
    BLOCK=`xfs_io -f -c "bmap" $DIR/test.$x | grep "0: \[" | sed
's/\../ /g' | cut -f5 -d" "`
    dd if=/dev/zero of=$DIR/test.$x bs=1M count=13 conv=notrunc,nocreat
    sync
done

where "allocnear" just creates a new file with the near block hint. It
isn't pretty atm but it does a better job of allocating files without
any block gaps between them. FYI the allocation patch is bypassed on
newer kernels and is useless without modification thanks to:

  http://www.mail-archive.com/ocfs2-devel@oss.oracle.com/msg04387.html

>> I'm still trying to get my head around how I would actually write a
>> userspace app/script to use it but I think it should allow me to do
>> what I want. It would be good if I could script it through xfs_io. I'd
>> really like a script where I could point it at a directory and it
>> would do something like:
>>
>>   1. count total space used by file sequence
>>   2. find start block for that much contiguous space on disk (or as
>> much of it as possible)
>>   3. allocate the files using the start block one after another on disk
>>
>>>> Another option might be to create a single contiguous large file,
>>>> concatenate all the images into it and then split it up on disk using
>>>> offsets but I don't think such a thing is even possible? I always know
>>>> the image sequence size beforehand, all images are exactly the same
>>>> size and I can control/freeze the filesystem access if needed.
>>>>
>>>> Anybody got any suggestions? It *seems* like something that should be
>>>> possible and would be useful.
>>> This would be pretty low-level control of the allocator by userspace.
>>>
>>> I'll just go back and ask what problem you're trying to solve?  There
>>> may be a better (i.e. currently existing) solution.
>>
>> The "realtime" option is sometimes suggested as a way to do sequence
>> streaming but I'd really rather avoid that. It seems to me like the
>> option to allocate a sequence of files end on end in a known chunk of
>> contiguous space is something that would be useful in the normal
>> operating mode.
>
> It would be, but it's not there now.  Also, without some more complexity
> it'd still probably end up being a best effort rather than a guarantee,
> but some hints from userspace might be better than nothing.

I'm pretty sure I can do what I need to do now. Just a case of writing
a userspace application to "defrag" a directory of images now ....

Thanks for the feedback,

Daire

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Matthias Schniedermeyer]

On 27.09.2010 17:30, Daire Byrne wrote:
> Eric,
> 
> On Mon, Sep 27, 2010 at 5:08 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> > Daire Byrne wrote:
> >>> Why is this the goal, what are you trying to achieve?
> >>
> >> I am essentially trying to play back a large frame sequence and trying
> >> to minimise seeks as it can lead to sporadic slowdowns on a SATA based
> >> RAID.
> >
> > Ok - and you've really seen allocation patterns that cause the playback
> > to slow down?  xfs_bmap information for a few sequential files that were
> > this far off would be interesting to see.
> >
> > Are you certain that it's seekiness causing the problem?  A great way
> > to visualize it would be to use the seekwatcher application while you
> > run a problematic file sequence.
> 
> I'm certain that the seekiness is the culprit. The image files are
> pretty big and require 400MB/s+ speeds to play them back at full rate.
> I can play a sequence which is aligned perfectly on disk just fine
> (readahead) but when seeks are required between frames the framerate
> drops noticeably. I'm using SATA disks which probably doesn't help
> matters.

As long as the disc-subsystem can sustain more than 400MB/s, there is a 
way to do "poor men's mega readahead".

I assume the sequence in which the files are accessed is predetermined 
and RAM is plentiful?

Then you can write a program/script that utilizes inotify to identify 
the file that is currently read and reads, say, 15 frames ahead, 
assuming that the sequence has 30fps the physical disc access is then 
about 1/2 a second ahead of time.

Of course the disc-subsystem has to be able to do bursts of more than 
400MB/s, so that when there is "stuttering" it can catch up before the 
"buffer runs empty".

A solution with a guarantee that the files stay in RAM could be done 
with a tmpfs. Depending on the playing program you may have to fake a 
little, but either sparse-files or symlinks should do the trick to have 
every file visible and replacing it with the real file/content a little 
before it is used and droping it afterwards.





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Roger Willcocks]

On Sun, 2010-09-26 at 23:08 -0500, Eric Sandeen wrote:
> Daire Byrne wrote:
> > Eric,
> > 
> > On Wed, Sep 22, 2010 at 9:10 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> >> Daire Byrne wrote:
> >>> Hi,
> >>>
> >>> I have been trying to figure out how to lay down a file sequence (e.g.
> >>> images) such that they are guaranteed to always be contiguous on disk
> >>> (i.e. no block gaps between them).
> >> There's no mechanism to guarantee that.
> >>
> >> Why is this the goal, what are you trying to achieve?
> > 
> > I am essentially trying to play back a large frame sequence and trying
> > to minimise seeks as it can lead to sporadic slowdowns on a SATA based
> > RAID.
> 
> Ok - and you've really seen allocation patterns that cause the playback
> to slow down?  xfs_bmap information for a few sequential files that were
> this far off would be interesting to see.

We had a similar requirement which is what prompted the original
allocate near patch. Our problem's not reading an individual file
sequence, but reading several sequences simultaneously. If the system's
only just keeping up reading some number of files from one sequence
alternating with some number from another, the additional overhead of
mid-sequence seeks can push you over the edge. So you need a mechanism
to maintain a separate 'write cursor' for each sequence, which (give or
take) lays the files out contiguously.

> >> I'll just go back and ask what problem you're trying to solve?  There
> >> may be a better (i.e. currently existing) solution.
> > 

I notice that commit 2a82b8be8a8dacb48cb7371449a7a9daa558b4a8 added
"Concurrent Multi-File Data Streams" to XFS which seems to do just about
the same thing. The comments therein explain the problem well. I've not
used it though.

 Also, without some more complexity
> it'd still probably end up being a best effort rather than a guarantee,
> but some hints from userspace might be better than nothing.

The 'allocate near' code simply hooks into the 'extend an existing file'
allocator, so it will generally do something sensible but as you say, it's
best effort not a guarantee. But it turns out that (here, at least) that's
good enough.

--
Roger




_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Contiguous file sequences

[Dave Chinner]

On Mon, Sep 27, 2010 at 05:30:35PM +0100, Daire Byrne wrote:
> Eric,
> 
> On Mon, Sep 27, 2010 at 5:08 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> > Daire Byrne wrote:
> >>> Why is this the goal, what are you trying to achieve?
> >>
> >> I am essentially trying to play back a large frame sequence and trying
> >> to minimise seeks as it can lead to sporadic slowdowns on a SATA based
> >> RAID.
> >
> > Ok - and you've really seen allocation patterns that cause the playback
> > to slow down?  xfs_bmap information for a few sequential files that were
> > this far off would be interesting to see.
> >
> > Are you certain that it's seekiness causing the problem?  A great way
> > to visualize it would be to use the seekwatcher application while you
> > run a problematic file sequence.
> 
> I'm certain that the seekiness is the culprit. The image files are
> pretty big and require 400MB/s+ speeds to play them back at full rate.

Ah, so you're doing ingest and real-time playback of 30fps uncompressed 2k
HD video streams?

> >>> You can't specify a starting block for any given file I'm afraid.
> >>
> >> Somebody pointed me at this which looks fairly promising:
> >>
> >>   http://oss.sgi.com/archives/xfs/2006-07/msg01005.html
> >
> > Yeah, that never got merged, but I think it still could be.
> >
> > It's only half your battle though, you need to find that contiguous
> > space first, then specify the start block for it with the interface
> > above.
> 
> I played around with the patch and I think I have a way to do what I
> want using something like:
> 
> # allocate a big file that all the frames can fit into and hope it is contiguous
> BLOCK=`xfs_io -f -c "resvsp 0 $TOTALSIZE" -c "freesp $FRAMESIZE 0" -c
> "pwrite 0 1" -c "bmap" $DIR/test.0 | grep "0: \[" | sed 's/\../ /g' |
> cut -f5 -d" "`
> for x in `seq 1 $FRAMES`; do
>     allocnear $DIR/test.$x $BLOCK
>     BLOCK=`xfs_io -f -c "bmap" $DIR/test.$x | grep "0: \[" | sed
> 's/\../ /g' | cut -f5 -d" "`
>     dd if=/dev/zero of=$DIR/test.$x bs=1M count=13 conv=notrunc,nocreat
>     sync
> done

I think you're doing it all wrong. You're using buffered IO, and
that is simply does not give control of the order of writeback of files
and hence where they might be allocated. Use direct IO, and you get
allocation occuring in the context of the write() syscall, and
if your application is single threaded, you'll see something like this:

$ for i in `seq 0 1 200`; do \
> dd if=/dev/zero of=/mnt/scratch/test.$i bs=1M count=13 oflag=direct
> done
......
$ for i in `seq 0 1 200`; do \
> sudo xfs_bmap -vp /mnt/scratch/test.$i |grep "0: \[";
> done
   0: [0..26623]:      96..26719         0 (96..26719)      26624 00000
   0: [0..26623]:      26720..53343      0 (26720..53343)   26624 00000
   0: [0..26623]:      53344..79967      0 (53344..79967)   26624 00000
   0: [0..26623]:      79968..106591     0 (79968..106591)  26624 00000
   0: [0..26623]:      106592..133215    0 (106592..133215) 26624 00000
   0: [0..26623]:      133216..159839    0 (133216..159839) 26624 00000
   0: [0..26623]:      159840..186463    0 (159840..186463) 26624 00000
   0: [0..26623]:      186464..213087    0 (186464..213087) 26624 00000
   0: [0..26623]:      213088..239711    0 (213088..239711) 26624 00000
   0: [0..26623]:      239712..266335    0 (239712..266335) 26624 00000
   0: [0..26623]:      266336..292959    0 (266336..292959) 26624 00000
   0: [0..26623]:      292968..319591    0 (292968..319591) 26624 00000
   0: [0..26623]:      319592..346215    0 (319592..346215) 26624 00000
   0: [0..26623]:      346216..372839    0 (346216..372839) 26624 00000
   0: [0..26623]:      372840..399463    0 (372840..399463) 26624 00000
   0: [0..26623]:      399464..426087    0 (399464..426087) 26624 00000
   0: [0..26623]:      426088..452711    0 (426088..452711) 26624 00000
   0: [0..26623]:      452712..479335    0 (452712..479335) 26624 00000
   0: [0..26623]:      479336..505959    0 (479336..505959) 26624 00000
   0: [0..26623]:      505960..532583    0 (505960..532583) 26624 00000
   0: [0..26623]:      532584..559207    0 (532584..559207) 26624 00000
   0: [0..26623]:      559208..585831    0 (559208..585831) 26624 00000
   0: [0..26623]:      585832..612455    0 (585832..612455) 26624 00000
   0: [0..26623]:      612456..639079    0 (612456..639079) 26624 00000
   0: [0..26623]:      639080..665703    0 (639080..665703) 26624 00000
   0: [0..26623]:      665704..692327    0 (665704..692327) 26624 00000
   0: [0..26623]:      692328..718951    0 (692328..718951) 26624 00000
   0: [0..26623]:      718952..745575    0 (718952..745575) 26624 00000
   0: [0..26623]:      745576..772199    0 (745576..772199) 26624 00000
   0: [0..26623]:      772200..798823    0 (772200..798823) 26624 00000 
.....

Looks pretty contiguous across files to me, and this was using
default mkfs and mount options. i.e. without needing preallocation,
hints or even the filestreams allocator.

FWIW, the filestreams allocator was designed to work optimally with
direct IO - it mostly works with buffered IO but you give up strict
ordering of allocation. That is, buffered IO does not strictly write
back files in exactly the same order that they were originally
written.

Further, the way you read the files using direct IO makes a very big
difference to performance. Reading them using 13x 1MB direct IOs:

$ time for i in `seq 0 1 200`; do \
> dd of=/dev/null if=/mnt/scratch/test.$i bs=1M count=13 iflag=direct;
> done
....
13+0 records in
13+0 records out
13631488 bytes (14 MB) copied, 0.12288 s, 111 MB/s

real    0m31.477s
user    0m0.276s
sys     0m0.628s

Which looks pretty bad considering the disk subsystem can do
1.6GB/s. However, even with buffered IO, the same read pattern could
not sustain 30fps uncompressed 2k video rates:

$ time for i in `seq 0 1 200`; do \
> dd of=/dev/null if=/mnt/scratch/test.$i bs=13M count=1;
> done
.....
1+0 records in
1+0 records out
13631488 bytes (14 MB) copied, 0.0649989 s, 210 MB/s

real    0m13.649s
user    0m0.072s
sys     0m4.100s

So you'd still need to do application level per-file readahead and
buffering.

However, being smart about direct IO, lets do a single 13MB IO per
frame:

$ time for i in `seq 0 1 200`; do \
> dd of=/dev/null if=/mnt/scratch/test.$i bs=13M count=1 iflag=direct;
> done
.....
1+0 records in
1+0 records out
13631488 bytes (14 MB) copied, 0.0211065 s, 646 MB/s

real    0m6.545s
user    0m0.044s
sys     0m1.808s

It's an awful lot faster with IO times being 3x lower than for
buffered IO. IOWs, you could probably play a video stream straight
off the disk without buffering or readahead....

IOWs, what I'm showing you here is that even with a disk subsystem
that does far in excess of your target throughput, the way you read
the files has a massive impact on IO latency. Even for perfect
layout, the above example shows that a single (optimal) direct IO
read has 3x lower IO latency than the same (optimal) buffered IO.
Direct Io is going to be a lot more deterministic, as well...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs