Re: XFS and DPX files

Eric Sandeen
2009-11-03T03:09:22Z

AndrewL733@aol.com wrote:
>
>>> I believe for a 15 drive RAID-6, where 2 disks are used
>>> forredundancy, the correct mkfs would be:
>>> mkfs -t xfs -d su=65536,sw=13 /dev/sdXX
>>>
>>
>> Yes you're right, I replied a bit too quickly :)
>>
>>
>>
>>> Another thing to try is if it would help to turn disk cache writes
>>> *on*, despite all warnings if the FAQ.
>
> Thank you for your suggestions. Yes I have write caching enabled. And I
> have StorSave set to "Performance". And I have a UPS on the system at
> all times!
>
> The information about barriers was useful. In years past I was running
> much older firmware for the 3ware 9650 cards and that did not support
> barriers. But it is true the current firmware does support barriers. I
> also believe the 3ware StorSave "Performance" setting will disable
> barriers as well -- at least it makes the card ignore FUA commands.
>
> Anyway, I have mounted the XFS filesystem with the "nobarrier" flag and
> I'm still seeing the same behavior. If you want to take a closer look
> at what I mean, please go to this link:
>
> http://sites.google.com/site/andrewl733info/xfs_and_dpx
>
> At this point, I have tried the following -- and none of these
> approaches seems to fix the problem:
>
> -- preallocation of DPX files
> -- reservation of DXP files (Make 10,000 zero-byte files named
> 0000001.dpx through 0010000.dpx)
> -- creating xfs filesystem with external log device (also a 16-drive
> RAID array, because that's what I have available)
> -- mounting with large logbsize
> -- mounting with more logbufs
> -- mounting with larger allocsize

Have you said how large the filesystem is? If it's > 1T or 2T, and
you're on a 64-bit system, have you tried the inode64 to get nicer inode
vs. data allocation behavior?

Other suggestions might be to try blktrace/seekwatcher to see where your
IO is going, or maybe even oprofile to see if xfs is burning cpu
searching for allocations, or somesuch ...

-Eric

>
> Again, I want to point out that I don't have any problem with the
> underlying RAID device. On Linux itself, I get Bonnie++ scores of
> around 740 MB/sec reading and 650 MB/sec writing, minimum. Over 10
> Gigabit Ethernet, I can write uncompressed HD streams (160 MB/sec) and I
> can read 2K DPX files (300+ MB/sec). DD shows similar results.
>
> My gut feeling is that XFS is falling over after creating a certain
> number of new files. Because the DPX format creates one file for every
> frame (30 files/sec), it's not really a video stream. It's really like
> making 30 photoshop files per second. It seems as if some resource that
> XFS needs is being used up after a certain number of files are created,
> and that it is very disruptive and costly to get more of that resource.
> Why ext3 and ext4 can keep going past 60,000 files and xfs falls over
> after 4000 or 5000 files, I do not understand.
>
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs