Re: XFS_IOC_RESVSP64 versus XFS_IOC_ALLOCSP64 with multiple threads

[Stewart Smith <stewart@mysql.com>]

[-- Attachment #1: Type: text/plain, Size: 3662 bytes --]

On Tue, 2006-11-14 at 11:04 +1100, Sam Vaughan wrote: 
> Those extents are curiously uniform, all 32kB in size.  The fact that  
> both files' extents are in AG 8 suggests that the two directories  
> ndb_1_fs and ndb_2_fs filled their original AGs and spilled out into  
> other ones, which is when the interference would have started.   
> Looking at the directory hierarchy in your last email, you might be  
> better off if you could add another directory for the datafiles and  
> undofiles to live in, so they don't end up sharing their AG with  
> other stuff in their parent directory.

I think this is typically what the QA guys do (to help keep their sanity
if anything). Perhaps we should have this in our "best practice"
documentation as well...

> > for the data and undo files, we're just not changing their size except
> > at creation time, so that's okay.
> 
> I'd assumed that these files were being continually grown.  If all  
> this is happening at creation time then it shouldn't be too hard to  
> make sure the files are cleanly allocated with just one extent.  Does  
> the following not work on your file system?
> 
> $ touch a b
> $ for file in a b; do
>  > xfs_io -c 'allocsp 1G 0' $file &
>  > done; wait
> [1] 12312
> [2] 12313
> [1]-  Done                    xfs_io -c 'allocsp 1G 0' $file
> [2]+  Done                    xfs_io -c 'allocsp 1G 0' $file
> $ xfs_bmap -v a b
> a:
> EXT: FILE-OFFSET      BLOCK-RANGE          AG AG-OFFSET               
> TOTAL
>     0: [0..2097151]:    231732008..233829159  6 (11968856..14066007)  
> 2097152
> b:
> EXT: FILE-OFFSET      BLOCK-RANGE          AG AG-OFFSET               
> TOTAL
>     0: [0..2097151]:    233829160..235926311  6 (14066008..16163159)  
> 2097152
> $

That works fine on my file systems (or, on my rather full and well
used /home, as well as it can).

We're opening the files with O_DIRECT (or, if not available or fails,
O_SYNC)



> >> Now in your case you're using different directories, so your files
> >> are probably OK at the start of day.  Once the AGs they start in fill
> >> up though, the files for both processes will start getting allocated
> >> from the next available AG.  At that point, allocations that started
> >> out looking like the first test above will end up looking like the
> >> second.
> >>
> >> The filestreams allocator will stop this from happening for
> >> applications that write data regularly like video ingest servers, but
> >> I wouldn't expect it to be a cure-all for your database app because
> >> your writes could have large delays between them.  Instead, I'd look
> >> into ways to break up your data into AG-sized chunks, starting a new
> >> directory every time you go over that magic size.
> >
> > I'll have to check our writing behaviour the files that change  
> > sizes...
> > but they're not too much of an issue (they're hardly ever read  
> > back, so
> > as long as writing them out is okay and reading isn't totally abismal,
> > we don't have to worry).
> 
> That's handy.  All in all it sounds like your requirements are very  
> file system friendly in terms of getting optimum allocation.  I'm not  
> sure what could be causing all those 32kB extents.

Perhaps being flushed out due to VM pressure? but with O_DIRECT/O_SYNC
that shouldn't be the case, right? Or perhaps *because* of
O_DIRECT/O_SYNC?
-- 
Stewart Smith, Software Engineer
MySQL AB, www.mysql.com
Office: +14082136540 Ext: 6616
VoIP: 6616@sip.us.mysql.com
Mobile: +61 4 3 8844 332

Jumpstart your cluster:
http://www.mysql.com/consulting/packaged/cluster.html

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]