Re: Contiguous file sequences

Roger Willcocks
2010-09-27T20:55:56Z

On Sun, 2010-09-26 at 23:08 -0500, Eric Sandeen wrote:
> Daire Byrne wrote:
> > Eric,
> >
> > On Wed, Sep 22, 2010 at 9:10 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> >> Daire Byrne wrote:
> >>> Hi,
> >>>
> >>> I have been trying to figure out how to lay down a file sequence (e.g.
> >>> images) such that they are guaranteed to always be contiguous on disk
> >>> (i.e. no block gaps between them).
> >> There's no mechanism to guarantee that.
> >>
> >> Why is this the goal, what are you trying to achieve?
> >
> > I am essentially trying to play back a large frame sequence and trying
> > to minimise seeks as it can lead to sporadic slowdowns on a SATA based
> > RAID.
>
> Ok - and you've really seen allocation patterns that cause the playback
> to slow down? xfs_bmap information for a few sequential files that were
> this far off would be interesting to see.

We had a similar requirement which is what prompted the original
allocate near patch. Our problem's not reading an individual file
sequence, but reading several sequences simultaneously. If the system's
only just keeping up reading some number of files from one sequence
alternating with some number from another, the additional overhead of
mid-sequence seeks can push you over the edge. So you need a mechanism
to maintain a separate 'write cursor' for each sequence, which (give or
take) lays the files out contiguously.

> >> I'll just go back and ask what problem you're trying to solve? There
> >> may be a better (i.e. currently existing) solution.
> >

I notice that commit 2a82b8be8a8dacb48cb7371449a7a9daa558b4a8 added
"Concurrent Multi-File Data Streams" to XFS which seems to do just about
the same thing. The comments therein explain the problem well. I've not
used it though.

Also, without some more complexity
> it'd still probably end up being a best effort rather than a guarantee,
> but some hints from userspace might be better than nothing.

The 'allocate near' code simply hooks into the 'extend an existing file'
allocator, so it will generally do something sensible but as you say, it's
best effort not a guarantee. But it turns out that (here, at least) that's
good enough.

--
Roger