* Re: RAID5 created by 8 disks works with xfs
2012-04-01 1:16 ` daobang wang
@ 2012-04-01 2:05 ` daobang wang
2012-04-01 5:13 ` Stan Hoeppner
2012-04-01 3:51 ` Stan Hoeppner
2012-04-01 4:52 ` Stan Hoeppner
2 siblings, 1 reply; 37+ messages in thread
From: daobang wang @ 2012-04-01 2:05 UTC (permalink / raw)
To: stan; +Cc: Mathias Burén, linux-raid
There is another problem occurred, it seems that the file system was
damaged when the pressure is very high, it reported input/output error
when i typed ls or other command, and I tried to repair it with
xfs_repair /dev/vg00/lv0000, the xfs_repir alloc memory failed, we
have 4GB memory on the machine, and the logical volume was a little
more than 15TB, Could it be repair successfully if we have enough
memory?
Thank you very much!
Best Wishes,
Daobang Wang.
On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
>
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
>
> 2. Volume Group on RAID5 with full capacity
>
> 3. Logical Volume on the Volume Group with full capacity
>
> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
>
> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.
>
> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal. is there any clue for optimize the application?
>
> Thank you very much.
>
> Best Regards,
> Daobang Wang.
>
> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 3/31/2012 2:59 AM, Mathias Burén wrote:
>>> On 31 March 2012 02:22, daobang wang <wangdb1981@gmail.com> wrote:
>>>> Hi ALL,
>>>>
>>>> How to adjust the xfs and raid parameters to improve the total
>>>> performance when RAID5 created by 8 disks works with xfs, and i writed
>>>> a test program, which started 100 threads to write big files, 500MB
>>>> per file, and delete it after writing finish. Thank you very much.
>>>>
>>>> Best Wishes,
>>>> Daobang Wang.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> Hi,
>>>
>>> See
>>> http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
>>> . Also see http://hep.kbfi.ee/index.php/IT/KernelTuning . For example,
>>> RAID5 with 8 harddrives and 64K stripe size:
>>>
>>> mkfs.xfs -d su=64k,sw=7 -l version=2,su=64k /dev/md0
>>
>> This is unnecessary. mkfs.xfs creates w/stripe alignment automatically
>> when the target device is an md device.
>>
>>> Consider mounting the filesystem with logbufs=8,logbsize=256k
>>
>> This is unnecessary for two reasons:
>>
>> 1. These are the default values in recent kernels
>> 2. His workload is the opposite of "metadata heavy"
>> logbufs and logbsize exist for metadata operations
>> to the journal, they are in memory journal write buffers
>>
>> The OP's stated workload is 100 streaming writes of 500MB files. This
>> is not anything close to a sane, real world workload. Writing 100 x
>> 500MB files in parallel to 7 spindles is an exercise in stupidity, and
>> especially to a RAID5 array with only 7 spindles. The OP is pushing
>> those drives to their seek limit of about 150 head seeks/sec without
>> actually writing much data, and *that* is what is ruining his
>> performance. What *should* be a streaming write workload of large files
>> has been turned into a massively random IO pattern due mostly to the
>> unrealistic write thread count, and partly to disk striping and the way
>> XFS allocation groups are created on a striped array.
>>
>> Assuming these are 2TB drives, to get much closer to ideal write
>> performance, and make this more of a streaming workload, what the OP
>> should be doing is writing no more than 8 files in parallel to at least
>> 8 different directories with XFS sitting on an md linear array of 4 md
>> RAID1 devices, assuming he needs protection from drive failure *and*
>> parallel write performance:
>>
>> $ mdadm -C /dev/md0 -l 1 -n 2 /dev/sd[ab]
>> $ mdadm -C /dev/md1 -l 1 -n 2 /dev/sd[cd]
>> $ mdadm -C /dev/md2 -l 1 -n 2 /dev/sd[ef]
>> $ mdadm -C /dev/md3 -l 1 -n 2 /dev/sd[gh]
>> $ mdadm -C /dev/md4 -l linear -n 4 /dev/md[0-3]
>> $ mkfs.xfs -d agcount=8 /dev/md4
>>
>> and mount with the inode64 option in fstab so we get the inode64
>> allocator, which spreads the metadata across all of the AGs instead of
>> stuffing in all in the first AG and yields other benefits.
>>
>> This setup eliminates striping, tons of head seeks, and gets much closer
>> to pure streaming write performance. Writing 8 files in parallel to 8
>> directories will cause XFS to put each file in a different allocation
>> group. Since we created 8 AGs, this means we'll have 2 files being
>> written to each disk in parallel. This reduces time wasted in head seek
>> latency by an order of magnitude and will dramatically increase disk
>> throughput in MB/s compared to the 100 files in parallel workload, which
>> again is simply stupid to do on this limited disk hardware.
>>
>> This 100 file parallel write workload needs about 6 times as many
>> spindles to be realistic, configured as a linear array of 24 RAID1
>> devices and formatted with 48 AGs. This would give you ~4 write streams
>> per drive, 2 per AG, or somewhere around 50% to 66% of the per drive
>> performance compared to the 8 drive 8 thread scenario I recommended
>> above.
>>
>> Final note: It is simply not possible to optimize XFS nor mdraid to get
>> you any better performance when writing 100 x 500MB files in parallel.
>> The lack of sufficient spindles is the problem.
>>
>> --
>> Stan
>>
>>
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 2:05 ` daobang wang
@ 2012-04-01 5:13 ` Stan Hoeppner
0 siblings, 0 replies; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-01 5:13 UTC (permalink / raw)
To: daobang wang; +Cc: Mathias Burén, linux-raid
On 3/31/2012 9:05 PM, daobang wang wrote:
> There is another problem occurred, it seems that the file system was
> damaged when the pressure is very high,
What kernel version are you using? Did you get an oops? What's in dmesg?
> it reported input/output error
Actual errors would be very helpful.
> when i typed ls or other command, and I tried to repair it with
> xfs_repair /dev/vg00/lv0000, the xfs_repir alloc memory failed, we
Did XFS automatically unmount the filesystem? If not, the error
reported may not indicate a problem with the filesystem. XFS shuts
filesystems down when it encounters serious problems.
> have 4GB memory on the machine, and the logical volume was a little
> more than 15TB, Could it be repair successfully if we have enough
> memory?
Hard to say. Depends on what happened and the extent of the damage, if
any. You've presented no log or debug information. I would think 4GB
should be plenty to run xfs_repair. Try
$ xfs_repair -n -vv -m 1 /dev/vg00/lv0000
dmem = in the output tells you how much RAM is needed for xfs_repair.
If it's more than 2GB and you're using a PAE kernel, switch to a 64 bit
kernel and 64 bit userland. If dmem is over 4GB then you need more
DIMMs in the machine. Or maybe simply dropping caches before running
xfs_repair might help:
$ echo 3 > /proc/sys/vm/drop_caches
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 1:16 ` daobang wang
2012-04-01 2:05 ` daobang wang
@ 2012-04-01 3:51 ` Stan Hoeppner
2012-04-01 5:12 ` daobang wang
2012-04-01 4:52 ` Stan Hoeppner
2 siblings, 1 reply; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-01 3:51 UTC (permalink / raw)
To: daobang wang; +Cc: Mathias Burén, linux-raid
On 3/31/2012 8:16 PM, daobang wang wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
>
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
>
> 2. Volume Group on RAID5 with full capacity
>
> 3. Logical Volume on the Volume Group with full capacity
LVM will create unnecessary overhead for this workload. Do not use it.
Directly format the md device with XFS and proper alignment. Again,
mkfs.xfs will do this automatically.
> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
What kernel version are you using? logbsize=262144 has been the default
for quite some time now.
NEVER disable barriers unless you have a quality BBWC RAID controller
and a good working UPS. Are these 8 disks connected to a BBWC RAID
card? Have you verified the write back cache is working properly?
> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.
This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
It is very likely that neither 8 nor 16 disks in RAID5 will be able to
sync this rate due to excessive head seeking, as I previously mentioned.
Having LVM layered between XFS and mdraid will make this situation even
worse.
> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal.
So now you have an aggregate 25 MB/s random write workload with 100
writers. This is still going to likely be too much for an 8 or 16 disk
RAID5 array for the same reason as 200 threads--too many disk seeks.
> is there any clue for optimize the application?
This is the Linux-raid list. We can help you optimize your RAID, and
since many of us use XFS, we can help you there as well. I've never
seen your application so it is impossible to suggest how to optimize it.
As far as optimizing your 16 disks with mdraid and XFS, I've already
given you pointers on how you need to configure your storage for optimal
performance with this workload. However, experience tells me you simply
don't have enough spindles to sync all these parallel writers if you
also need redundancy.
The fact that you're disabling barriers likely in lieu of BBWC seems to
indicate you're not terribly worried about losing all your data in the
event of a drive failure. If this is the case, 16 x 7.2k spindles with
6 parallel writers per spindle at .25 MB/s each might work. Simply
create an md linear array with 16 disks, no LVM, and format it with
$ mkfs.xfs -d agcount=48 /dev/md0
and configure you application to write each video stream file to a
different directory. This will allow for 96 total streams, 6 being
written to each drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 3:51 ` Stan Hoeppner
@ 2012-04-01 5:12 ` daobang wang
2012-04-01 5:40 ` Stan Hoeppner
0 siblings, 1 reply; 37+ messages in thread
From: daobang wang @ 2012-04-01 5:12 UTC (permalink / raw)
To: stan; +Cc: Mathias Burén, linux-raid
Thank you very much!
I got it, so we can remove the Volume Group and Logical Volume to save resource.
And i will try RAID5 with 16 disks to write 96 total streams again.
I used the Linux kernel 2.6.26.4.
And we do not have BBWC
The application has 16kb cache per stream, Is it possible to optimize
it if we use 32kb or 64kb cache?
On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 3/31/2012 8:16 PM, daobang wang wrote:
>> Thanks to Mathias and stan, Here is the detail of the configuration.
>>
>> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
>> the RAID5 created with Chunk Size of 64K and left-symmetric
>>
>> 2. Volume Group on RAID5 with full capacity
>>
>> 3. Logical Volume on the Volume Group with full capacity
>
> LVM will create unnecessary overhead for this workload. Do not use it.
> Directly format the md device with XFS and proper alignment. Again,
> mkfs.xfs will do this automatically.
>
>> 4. XFS filesystem created on the Logical Volume with option "-f -i
>> size=512", and mount option is "-t xfs -o
>> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
>
> What kernel version are you using? logbsize=262144 has been the default
> for quite some time now.
>
> NEVER disable barriers unless you have a quality BBWC RAID controller
> and a good working UPS. Are these 8 disks connected to a BBWC RAID
> card? Have you verified the write back cache is working properly?
>
>> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
>> files on the XFS.
>
> This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
> It is very likely that neither 8 nor 16 disks in RAID5 will be able to
> sync this rate due to excessive head seeking, as I previously mentioned.
> Having LVM layered between XFS and mdraid will make this situation even
> worse.
>
>> This is the pressure testing, just verify the reliability of the
>> system, we will not use it in real envrionment, 100 video streams
>> writen is our goal.
>
> So now you have an aggregate 25 MB/s random write workload with 100
> writers. This is still going to likely be too much for an 8 or 16 disk
> RAID5 array for the same reason as 200 threads--too many disk seeks.
>
>> is there any clue for optimize the application?
>
> This is the Linux-raid list. We can help you optimize your RAID, and
> since many of us use XFS, we can help you there as well. I've never
> seen your application so it is impossible to suggest how to optimize it.
> As far as optimizing your 16 disks with mdraid and XFS, I've already
> given you pointers on how you need to configure your storage for optimal
> performance with this workload. However, experience tells me you simply
> don't have enough spindles to sync all these parallel writers if you
> also need redundancy.
>
> The fact that you're disabling barriers likely in lieu of BBWC seems to
> indicate you're not terribly worried about losing all your data in the
> event of a drive failure. If this is the case, 16 x 7.2k spindles with
> 6 parallel writers per spindle at .25 MB/s each might work. Simply
> create an md linear array with 16 disks, no LVM, and format it with
>
> $ mkfs.xfs -d agcount=48 /dev/md0
>
> and configure you application to write each video stream file to a
> different directory. This will allow for 96 total streams, 6 being
> written to each drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive.
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 5:12 ` daobang wang
@ 2012-04-01 5:40 ` Stan Hoeppner
2012-04-01 5:59 ` daobang wang
2012-04-01 10:33 ` David Brown
0 siblings, 2 replies; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-01 5:40 UTC (permalink / raw)
To: daobang wang; +Cc: Mathias Burén, linux-raid
On 4/1/2012 12:12 AM, daobang wang wrote:
> Thank you very much!
> I got it, so we can remove the Volume Group and Logical Volume to save resource.
> And i will try RAID5 with 16 disks to write 96 total streams again.
Why do you keep insisting on RAID5?!?! It is not suitable for your
workload. It sucks Monday through Saturday and twice on Sunday for this
workload.
Test your 16 drive RAID5 array head to head with the linear array + XFS
architecture I gave you instructions to create, and report back your
results.
> I used the Linux kernel 2.6.26.4.
Which distro?
2.6.26 is *ancient* and has storage layer bugs. It does NOT have
delaylog, which was introduced in 2.6.35, and wasn't fully performant
until 2.6.38+.
You're building and testing a new platform with a terribly obsolete
distribution. You need a much newer kernel and distro. 3.0.x would be
best. Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
> And we do not have BBWC
Then you must re-enable barriers or perennially suffer more filesystem
problems. I simply cannot emphasize enough how critical write barriers
are to filesystem consistency.
> The application has 16kb cache per stream, Is it possible to optimize
> it if we use 32kb or 64kb cache?
No. Read the app's documentation. This caching is to prevent dropped
frames in the recorded file. Increasing this value won't affect disk
performance.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 5:40 ` Stan Hoeppner
@ 2012-04-01 5:59 ` daobang wang
2012-04-01 6:20 ` daobang wang
2012-04-01 10:33 ` David Brown
1 sibling, 1 reply; 37+ messages in thread
From: daobang wang @ 2012-04-01 5:59 UTC (permalink / raw)
To: stan; +Cc: Mathias Burén, linux-raid
So sorry, the kernel version should be 2.6.36.4, and we do not use
distro, we compiled the kernel codes and user space codes.
I'm duplicating the input/output error issue, the system was
restarted, I will dump the dmesg log if i can duplicate it.
Thanks again,
Daobang Wang.
On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/1/2012 12:12 AM, daobang wang wrote:
>> Thank you very much!
>> I got it, so we can remove the Volume Group and Logical Volume to save
>> resource.
>> And i will try RAID5 with 16 disks to write 96 total streams again.
>
> Why do you keep insisting on RAID5?!?! It is not suitable for your
> workload. It sucks Monday through Saturday and twice on Sunday for this
> workload.
>
> Test your 16 drive RAID5 array head to head with the linear array + XFS
> architecture I gave you instructions to create, and report back your
> results.
>
>> I used the Linux kernel 2.6.26.4.
>
> Which distro?
>
> 2.6.26 is *ancient* and has storage layer bugs. It does NOT have
> delaylog, which was introduced in 2.6.35, and wasn't fully performant
> until 2.6.38+.
>
> You're building and testing a new platform with a terribly obsolete
> distribution. You need a much newer kernel and distro. 3.0.x would be
> best. Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>
>> And we do not have BBWC
>
> Then you must re-enable barriers or perennially suffer more filesystem
> problems. I simply cannot emphasize enough how critical write barriers
> are to filesystem consistency.
>
>> The application has 16kb cache per stream, Is it possible to optimize
>> it if we use 32kb or 64kb cache?
>
> No. Read the app's documentation. This caching is to prevent dropped
> frames in the recorded file. Increasing this value won't affect disk
> performance.
>
> --
> Stan
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 5:59 ` daobang wang
@ 2012-04-01 6:20 ` daobang wang
2012-04-01 7:08 ` Marcus Sorensen
2012-04-02 3:12 ` Stan Hoeppner
0 siblings, 2 replies; 37+ messages in thread
From: daobang wang @ 2012-04-01 6:20 UTC (permalink / raw)
To: stan; +Cc: Mathias Burén, linux-raid
I have the different opinion, the application does not write the disk
directly, disk IOs will be merged before writen in kernel, just we can
not caculate how many IOs will be merged.
On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
> So sorry, the kernel version should be 2.6.36.4, and we do not use
> distro, we compiled the kernel codes and user space codes.
>
> I'm duplicating the input/output error issue, the system was
> restarted, I will dump the dmesg log if i can duplicate it.
>
> Thanks again,
> Daobang Wang.
>
> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 4/1/2012 12:12 AM, daobang wang wrote:
>>> Thank you very much!
>>> I got it, so we can remove the Volume Group and Logical Volume to save
>>> resource.
>>> And i will try RAID5 with 16 disks to write 96 total streams again.
>>
>> Why do you keep insisting on RAID5?!?! It is not suitable for your
>> workload. It sucks Monday through Saturday and twice on Sunday for this
>> workload.
>>
>> Test your 16 drive RAID5 array head to head with the linear array + XFS
>> architecture I gave you instructions to create, and report back your
>> results.
>>
>>> I used the Linux kernel 2.6.26.4.
>>
>> Which distro?
>>
>> 2.6.26 is *ancient* and has storage layer bugs. It does NOT have
>> delaylog, which was introduced in 2.6.35, and wasn't fully performant
>> until 2.6.38+.
>>
>> You're building and testing a new platform with a terribly obsolete
>> distribution. You need a much newer kernel and distro. 3.0.x would be
>> best. Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>>
>>> And we do not have BBWC
>>
>> Then you must re-enable barriers or perennially suffer more filesystem
>> problems. I simply cannot emphasize enough how critical write barriers
>> are to filesystem consistency.
>>
>>> The application has 16kb cache per stream, Is it possible to optimize
>>> it if we use 32kb or 64kb cache?
>>
>> No. Read the app's documentation. This caching is to prevent dropped
>> frames in the recorded file. Increasing this value won't affect disk
>> performance.
>>
>> --
>> Stan
>>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 6:20 ` daobang wang
@ 2012-04-01 7:08 ` Marcus Sorensen
2012-04-02 3:47 ` Stan Hoeppner
2012-04-02 3:12 ` Stan Hoeppner
1 sibling, 1 reply; 37+ messages in thread
From: Marcus Sorensen @ 2012-04-01 7:08 UTC (permalink / raw)
To: daobang wang; +Cc: stan, Mathias Burén, linux-raid
Streaming workloads don't benefit much from writeback cache.
Writeback can absorb spikes, but if you have a constant load that goes
beyond what your disks can handle, you'll have good performance
exactly to the point where your writeback is full. Once you hit
dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
with I/O beyond recovery. It's best to limit your writeback cache to a
relatively small number with such a constant IO load.
You are right that merging could help to some degree, but you likely
won't be merging I/Os from separate streams, so your workload is still
terribly random and you just end up with larger random I/Os. I don't
think it will make up for the difference between your workload and
your configuration.
On Sun, Apr 1, 2012 at 12:20 AM, daobang wang <wangdb1981@gmail.com> wrote:
> I have the different opinion, the application does not write the disk
> directly, disk IOs will be merged before writen in kernel, just we can
> not caculate how many IOs will be merged.
>
> On 4/1/12, daobang wang <wangdb1981@gmail.com> wrote:
>> So sorry, the kernel version should be 2.6.36.4, and we do not use
>> distro, we compiled the kernel codes and user space codes.
>>
>> I'm duplicating the input/output error issue, the system was
>> restarted, I will dump the dmesg log if i can duplicate it.
>>
>> Thanks again,
>> Daobang Wang.
>>
>> On 4/1/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>> On 4/1/2012 12:12 AM, daobang wang wrote:
>>>> Thank you very much!
>>>> I got it, so we can remove the Volume Group and Logical Volume to save
>>>> resource.
>>>> And i will try RAID5 with 16 disks to write 96 total streams again.
>>>
>>> Why do you keep insisting on RAID5?!?! It is not suitable for your
>>> workload. It sucks Monday through Saturday and twice on Sunday for this
>>> workload.
>>>
>>> Test your 16 drive RAID5 array head to head with the linear array + XFS
>>> architecture I gave you instructions to create, and report back your
>>> results.
>>>
>>>> I used the Linux kernel 2.6.26.4.
>>>
>>> Which distro?
>>>
>>> 2.6.26 is *ancient* and has storage layer bugs. It does NOT have
>>> delaylog, which was introduced in 2.6.35, and wasn't fully performant
>>> until 2.6.38+.
>>>
>>> You're building and testing a new platform with a terribly obsolete
>>> distribution. You need a much newer kernel and distro. 3.0.x would be
>>> best. Debian 6.0.4 with a backport 3.0.x kernel would be a good start.
>>>
>>>> And we do not have BBWC
>>>
>>> Then you must re-enable barriers or perennially suffer more filesystem
>>> problems. I simply cannot emphasize enough how critical write barriers
>>> are to filesystem consistency.
>>>
>>>> The application has 16kb cache per stream, Is it possible to optimize
>>>> it if we use 32kb or 64kb cache?
>>>
>>> No. Read the app's documentation. This caching is to prevent dropped
>>> frames in the recorded file. Increasing this value won't affect disk
>>> performance.
>>>
>>> --
>>> Stan
>>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 7:08 ` Marcus Sorensen
@ 2012-04-02 3:47 ` Stan Hoeppner
2012-04-05 0:48 ` daobang wang
0 siblings, 1 reply; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-02 3:47 UTC (permalink / raw)
To: Marcus Sorensen; +Cc: daobang wang, Mathias Burén, linux-raid
On 4/1/2012 2:08 AM, Marcus Sorensen wrote:
> Streaming workloads don't benefit much from writeback cache.
> Writeback can absorb spikes, but if you have a constant load that goes
> beyond what your disks can handle, you'll have good performance
> exactly to the point where your writeback is full. Once you hit
> dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
> with I/O beyond recovery. It's best to limit your writeback cache to a
> relatively small number with such a constant IO load.
My comments WRT battery or flash backed write cache, whether write back
or write through, were strictly related to running with XFS barriers
disabled. The only scenaio where you can safely disable XFS barriers is
when you have a properly functioning BBWC RAID controller, whether an
HBA, or a host independent external array such as a SAN controller.
Of course, I agree 100% that write cache yields little benefit with high
throughput workloads, and especially those generating high seek rates to
boot. The workload described is many parallel streaming writes of .25
MB/s each. If we use 96 streams, that's "only" 24 MB/s aggregate. But
as each of the 16 drives will likely be hitting its seek ceiling of
~150/s using XFS on striped RAID, the aggregate throughput of the 15
RAID5 spindles will probably be less than 10 MB/s.
Using the linear array with XFS instead of RAID5 will eliminate much of
the head seeking, increasing throughput. The increase may not be a
huge, but it will be enough to handle many more parallel write streams
than RAID5 before the drives hit their seek ceiling.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-02 3:47 ` Stan Hoeppner
@ 2012-04-05 0:48 ` daobang wang
[not found] ` <CACwgYDOtCoVF-p+KKqPYxHhA4vWF78Ueecx9hcVWLoyxFWzV9Q@mail.gmail.com>
0 siblings, 1 reply; 37+ messages in thread
From: daobang wang @ 2012-04-05 0:48 UTC (permalink / raw)
To: stan; +Cc: Marcus Sorensen, Mathias Burén, linux-raid
Thanks to all. Because the end-user is adhere to use RAID5, I need to
test how many video streams could be run on it really, and give the
suggestion to them. Thanks again.
Best Wishes,
Daobang Wang.
On 4/2/12, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 4/1/2012 2:08 AM, Marcus Sorensen wrote:
>> Streaming workloads don't benefit much from writeback cache.
>> Writeback can absorb spikes, but if you have a constant load that goes
>> beyond what your disks can handle, you'll have good performance
>> exactly to the point where your writeback is full. Once you hit
>> dirty_bytes, dirty_ratio, or the timeout, your system will be crushed
>> with I/O beyond recovery. It's best to limit your writeback cache to a
>> relatively small number with such a constant IO load.
>
> My comments WRT battery or flash backed write cache, whether write back
> or write through, were strictly related to running with XFS barriers
> disabled. The only scenaio where you can safely disable XFS barriers is
> when you have a properly functioning BBWC RAID controller, whether an
> HBA, or a host independent external array such as a SAN controller.
>
> Of course, I agree 100% that write cache yields little benefit with high
> throughput workloads, and especially those generating high seek rates to
> boot. The workload described is many parallel streaming writes of .25
> MB/s each. If we use 96 streams, that's "only" 24 MB/s aggregate. But
> as each of the 16 drives will likely be hitting its seek ceiling of
> ~150/s using XFS on striped RAID, the aggregate throughput of the 15
> RAID5 spindles will probably be less than 10 MB/s.
>
> Using the linear array with XFS instead of RAID5 will eliminate much of
> the head seeking, increasing throughput. The increase may not be a
> huge, but it will be enough to handle many more parallel write streams
> than RAID5 before the drives hit their seek ceiling.
>
> --
> Stan
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 6:20 ` daobang wang
2012-04-01 7:08 ` Marcus Sorensen
@ 2012-04-02 3:12 ` Stan Hoeppner
1 sibling, 0 replies; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-02 3:12 UTC (permalink / raw)
To: daobang wang; +Cc: Mathias Burén, linux-raid
On 4/1/2012 1:20 AM, daobang wang wrote:
> I have the different opinion, the application does not write the disk
> directly, disk IOs will be merged before writen in kernel, just we can
> not caculate how many IOs will be merged.
Your original workload description was writing 200 or 100 video streams
to 200 or 100 different files in parallel. What you state above is
different than the original. Maybe you could give us a little more
detail about how this application actually does its file IO.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 5:40 ` Stan Hoeppner
2012-04-01 5:59 ` daobang wang
@ 2012-04-01 10:33 ` David Brown
2012-04-01 12:28 ` John Robinson
2012-04-02 5:43 ` Stan Hoeppner
1 sibling, 2 replies; 37+ messages in thread
From: David Brown @ 2012-04-01 10:33 UTC (permalink / raw)
To: linux-raid
On 01/04/12 07:40, Stan Hoeppner wrote:
> On 4/1/2012 12:12 AM, daobang wang wrote:
>> Thank you very much!
>> I got it, so we can remove the Volume Group and Logical Volume to save resource.
>> And i will try RAID5 with 16 disks to write 96 total streams again.
>
> Why do you keep insisting on RAID5?!?! It is not suitable for your
> workload. It sucks Monday through Saturday and twice on Sunday for this
> workload.
>
My thoughts on this setup are that RAID5 (or RAID6) is a poor choice -
it can quickly cause a mess for streaming writes. Even if the streams
themselves can mostly end up as full stripe writes, odd writes such as
the end of a file, metadata writes, log data writes, etc., will mean
read-modify-write operations that will cripple the write performance for
the rest of the operations.
And if a disk fails so that you are running degraded, it will be hopeless.
Either drop the redundancy requirement entirely (maybe by making sure
other backups are in order), or double the spindles and use RAID1 / RAID10.
For an application like this, it would probably make sense to put the
xfs log (and the mdraid bitmap file, if you are using one) on a separate
disk - perhaps a small SSD.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 10:33 ` David Brown
@ 2012-04-01 12:28 ` John Robinson
2012-04-02 6:59 ` David Brown
[not found] ` <CA+res+QkLi7sxZrD-XOcbR47CeJ5gADf7P6pa1w1oMf8CKSB4g@mail.gmail.com>
2012-04-02 5:43 ` Stan Hoeppner
1 sibling, 2 replies; 37+ messages in thread
From: John Robinson @ 2012-04-01 12:28 UTC (permalink / raw)
To: linux-raid
On 01/04/2012 11:33, David Brown wrote:
[...]
> For an application like this, it would probably make sense to put the
> xfs log (and the mdraid bitmap file, if you are using one) on a separate
> disk - perhaps a small SSD.
Now that you've suggested that, it occurs to me that for an application
like this, the OP might be better doing his multiple slow streams to a
spool folder on SSD, and copying over to the big array of spinning rust
when each stream completes. 200 streams of 500MB is 100GB of data, so a
pair of slightly larger SSDs in RAID1 (or RAID10, to balance the reads
coming off) would do nicely as a spool area.
This might also be a good application for bcache, FlashCache or whatever.
Cheers,
John.
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: RAID5 created by 8 disks works with xfs
2012-04-01 12:28 ` John Robinson
@ 2012-04-02 6:59 ` David Brown
[not found] ` <CA+res+QkLi7sxZrD-XOcbR47CeJ5gADf7P6pa1w1oMf8CKSB4g@mail.gmail.com>
1 sibling, 0 replies; 37+ messages in thread
From: David Brown @ 2012-04-02 6:59 UTC (permalink / raw)
To: linux-raid
On 01/04/2012 14:28, John Robinson wrote:
> On 01/04/2012 11:33, David Brown wrote:
> [...]
>> For an application like this, it would probably make sense to put the
>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>> disk - perhaps a small SSD.
>
> Now that you've suggested that, it occurs to me that for an application
> like this, the OP might be better doing his multiple slow streams to a
> spool folder on SSD, and copying over to the big array of spinning rust
> when each stream completes. 200 streams of 500MB is 100GB of data, so a
> pair of slightly larger SSDs in RAID1 (or RAID10, to balance the reads
> coming off) would do nicely as a spool area.
If the OP wants to do that, he must be careful when looking at the rated
write speeds for the SSDs. SSDs need a bit of a break on occasion to
get time to do their garbage collection - their long-term streamed write
speed is therefore lower than their short-term write speeds (and that's
the figure that will be given in the spec). But if you take that into
account, and short-partition the SSD's a little (i.e., leave 10%
unpartitioned, and put the raid1 on the partition rather than the disk)
so that there are always plenty of free blocks when they are needed,
that could be a very good solution.
>
> This might also be a good application for bcache, FlashCache or whatever.
>
> Cheers,
>
> John.
>
^ permalink raw reply [flat|nested] 37+ messages in thread[parent not found: <CA+res+QkLi7sxZrD-XOcbR47CeJ5gADf7P6pa1w1oMf8CKSB4g@mail.gmail.com>]
* Re: RAID5 created by 8 disks works with xfs
[not found] ` <CA+res+QkLi7sxZrD-XOcbR47CeJ5gADf7P6pa1w1oMf8CKSB4g@mail.gmail.com>
@ 2012-04-02 8:01 ` John Robinson
2012-04-02 10:01 ` Jack Wang
0 siblings, 1 reply; 37+ messages in thread
From: John Robinson @ 2012-04-02 8:01 UTC (permalink / raw)
To: Jack Wang; +Cc: linux-raid
On 02/04/2012 04:15, Jack Wang wrote:
> 2012/4/1 John Robinson <john.robinson@anonymous.org.uk
> <mailto:john.robinson@anonymous.org.uk>>
[...]
> Now that you've suggested that, it occurs to me that for an
> application like this, the OP might be better doing his multiple
> slow streams to a spool folder on SSD, and copying over to the big
> array of spinning rust when each stream completes. 200 streams of
> 500MB is 100GB of data, so a pair of slightly larger SSDs in RAID1
> (or RAID10, to balance the reads coming off) would do nicely as a
> spool area.
>
> This might also be a good application for bcache, FlashCache or
> whatever.
>
> Using bcache/flashcache is what I'm considering, does above
> configuration option still needed?
Using a pair of SSDs? I think if you're adding a cache to a redundant
array, you ought to make the cache redundant too.
Using RAID10 to balance reads coming off the redundant pair of SSDs?
That would depend on how bcache or flashcache is written; if it's
single-threaded or does large copies from cache to the backing, then
yes, you may see better performance or at least more balanced usage of
the two SSDs, and surely won't see worse performance.
Cheers,
John.
^ permalink raw reply [flat|nested] 37+ messages in thread* Re: RAID5 created by 8 disks works with xfs
2012-04-02 8:01 ` John Robinson
@ 2012-04-02 10:01 ` Jack Wang
2012-04-02 10:28 ` John Robinson
0 siblings, 1 reply; 37+ messages in thread
From: Jack Wang @ 2012-04-02 10:01 UTC (permalink / raw)
To: John Robinson; +Cc: linux-raid
2012/4/2 John Robinson <john.robinson@anonymous.org.uk>
>
> On 02/04/2012 04:15, Jack Wang wrote:
>>
>> 2012/4/1 John Robinson <john.robinson@anonymous.org.uk
>> <mailto:john.robinson@anonymous.org.uk>>
>
> [...]
>>
>> Now that you've suggested that, it occurs to me that for an
>> application like this, the OP might be better doing his multiple
>> slow streams to a spool folder on SSD, and copying over to the big
>> array of spinning rust when each stream completes. 200 streams of
>> 500MB is 100GB of data, so a pair of slightly larger SSDs in RAID1
>> (or RAID10, to balance the reads coming off) would do nicely as a
>> spool area.
>>
>> This might also be a good application for bcache, FlashCache or
>> whatever.
>>
>> Using bcache/flashcache is what I'm considering, does above
>> configuration option still needed?
>
>
> Using a pair of SSDs? I think if you're adding a cache to a redundant array, you ought to make the cache redundant too.
>
> Using RAID10 to balance reads coming off the redundant pair of SSDs? That would depend on how bcache or flashcache is written; if it's single-threaded or does large copies from cache to the backing, then yes, you may see better performance or at least more balanced usage of the two SSDs, and surely won't see worse performance.
>
> Cheers,
>
> John.
John,
Thanks for reply , I mean using SSD as cache device, and raid5
16*7200rpm disks, is this works for streaming workload above?
I know bcache is try to turn rand write to seqencial write, but most
ssd is not good at RAND write as I know.
Cheers,
Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-02 10:01 ` Jack Wang
@ 2012-04-02 10:28 ` John Robinson
2012-04-02 20:41 ` Stan Hoeppner
0 siblings, 1 reply; 37+ messages in thread
From: John Robinson @ 2012-04-02 10:28 UTC (permalink / raw)
To: Jack Wang; +Cc: linux-raid
On 02/04/2012 11:01, Jack Wang wrote:
> 2012/4/2 John Robinson<john.robinson@anonymous.org.uk>
>> On 02/04/2012 04:15, Jack Wang wrote:
>>> 2012/4/1 John Robinson<john.robinson@anonymous.org.uk
>>> <mailto:john.robinson@anonymous.org.uk>>
>> [...]
>>> This might also be a good application for bcache, FlashCache or
>>> whatever.
>>>
>>> Using bcache/flashcache is what I'm considering, does above
>>> configuration option still needed?
[...]
> Thanks for reply , I mean using SSD as cache device, and raid5
> 16*7200rpm disks, is this works for streaming workload above?
>
> I know bcache is try to turn rand write to seqencial write, but most
> ssd is not good at RAND write as I know.
I don't know where you got that from; SSDs are absolutely brilliant at
small random writes and reads, hundreds of times better than spinning
hard discs. bcache and flashcache were written specifically to take
advantage of SSDs' high I/Os per second for small (random) reads and writes.
I wouldn't touch RAID5 for anything other than 3 drives; I'd use RAID6
or RAID60 for 4-plus where I didn't need the better small/random write
performance of RAID10. I haven't actually used bcache or flashcache,
which is why I introduced them with "might".
My fantasy configuration in your 16-drive chassis would be 2 6-drive
RAID6s, striped together in RAID0, with an SSD cache over the top built
from two SSDs in RAID10 (or if I was feeling really paranoid, 3 SSDs in
RAID10,n3), with the remaining slot containing a hot spare for the RAID6s.
Cheers,
John.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-02 10:28 ` John Robinson
@ 2012-04-02 20:41 ` Stan Hoeppner
0 siblings, 0 replies; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-02 20:41 UTC (permalink / raw)
To: John Robinson; +Cc: Jack Wang, linux-raid
On 4/2/2012 5:28 AM, John Robinson wrote:
> My fantasy configuration in your 16-drive chassis would be 2 6-drive
> RAID6s, striped together in RAID0, with an SSD cache over the top built
> from two SSDs in RAID10 (or if I was feeling really paranoid, 3 SSDs in
> RAID10,n3), with the remaining slot containing a hot spare for the RAID6s.
This advice is not sound. The introductory "My fantasy" tends to put an
exclamation point on this fact. This configuration is completely
unsuitable for the workload described.
I already described the most optimal configuration for this workload,
given the OP's current hardware, including all the commands to build it.
The staged SSD idea has merit, as well as the SSD write cache idea. If
going this route, two SLC SSDs should simply be mirrored. Using 3 of
them or a RAID 1E variant on two of them is silly.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 10:33 ` David Brown
2012-04-01 12:28 ` John Robinson
@ 2012-04-02 5:43 ` Stan Hoeppner
2012-04-02 7:04 ` David Brown
1 sibling, 1 reply; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-02 5:43 UTC (permalink / raw)
To: David Brown; +Cc: linux-raid
On 4/1/2012 5:33 AM, David Brown wrote:
> For an application like this, it would probably make sense to put the
> xfs log (and the mdraid bitmap file, if you are using one) on a separate
> disk - perhaps a small SSD.
XFS only journals metadata changes. Thus an external journal is not
needed here as there is no metadata in the workload.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-02 5:43 ` Stan Hoeppner
@ 2012-04-02 7:04 ` David Brown
2012-04-02 20:21 ` Stan Hoeppner
0 siblings, 1 reply; 37+ messages in thread
From: David Brown @ 2012-04-02 7:04 UTC (permalink / raw)
To: stan; +Cc: David Brown, linux-raid
On 02/04/2012 07:43, Stan Hoeppner wrote:
> On 4/1/2012 5:33 AM, David Brown wrote:
>
>> For an application like this, it would probably make sense to put the
>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>> disk - perhaps a small SSD.
>
> XFS only journals metadata changes. Thus an external journal is not
> needed here as there is no metadata in the workload.
>
Won't there still be metadata changes as each file is closed and a new
one started, or as more space is allocated to the files being written (I
know xfs has delayed allocation and other tricks to minimise this, but
it can't be eliminated entirely). Each time the metadata or log needs
written, you'll get a big discontinuity in the writing as the disks
heads need to jump around (especially for RAID5!).
It would make sense to have a separate disk (again, SSD is logical) for
the OS and other files anyway, keeping the big array for the data.
Putting the xfs log there too would surely be a small but helpful step?
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-02 7:04 ` David Brown
@ 2012-04-02 20:21 ` Stan Hoeppner
0 siblings, 0 replies; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-02 20:21 UTC (permalink / raw)
To: David Brown; +Cc: David Brown, linux-raid
On 4/2/2012 2:04 AM, David Brown wrote:
> On 02/04/2012 07:43, Stan Hoeppner wrote:
>> On 4/1/2012 5:33 AM, David Brown wrote:
>>
>>> For an application like this, it would probably make sense to put the
>>> xfs log (and the mdraid bitmap file, if you are using one) on a separate
>>> disk - perhaps a small SSD.
>>
>> XFS only journals metadata changes. Thus an external journal is not
>> needed here as there is no metadata in the workload.
>>
>
> Won't there still be metadata changes as each file is closed and a new
> one started,
Yes, creating one file requires one journal operation consisting of a
few dozen to a few hundred bytes of journal data. With delaylog this
will very likely be consolidated into a single IO to the journal, or a
handful of IOs. With the workload in question the number of journal IOs
is trivial.
> or as more space is allocated to the files being written (I
Allocation occurs once when the file is created.
> know xfs has delayed allocation and other tricks to minimise this, but
> it can't be eliminated entirely). Each time the metadata or log needs
> written, you'll get a big discontinuity in the writing as the disks
> heads need to jump around (especially for RAID5!).
The ratio of file IOs to metadata IOs in this workload is well north of
millions to one. Again, the metadata IOPS in this workload are trivial.
RAID level has no bearing on metadata IOPS here. Though it does for
the actual workload, as I already discussed in detail.
> It would make sense to have a separate disk (again, SSD is logical) for
> the OS and other files anyway, keeping the big array for the data.
> Putting the xfs log there too would surely be a small but helpful step?
No it doesn't make sense. See above. Additionally, an external journal
can only be used if the journal device is behind battery backed RAID
cache. Due to a design issue, XFS cannot perform barrier operations on
an external journal device, only to an internal journal. Without
barriers or BBWC you have the opportunity for constant filesystem
corruption. The OP doesn't have a BBWC RAID controller, nor any RAID
controller at all. So an external journal isn't an option here anyway.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 1:16 ` daobang wang
2012-04-01 2:05 ` daobang wang
2012-04-01 3:51 ` Stan Hoeppner
@ 2012-04-01 4:52 ` Stan Hoeppner
2012-04-01 8:06 ` John Robinson
2 siblings, 1 reply; 37+ messages in thread
From: Stan Hoeppner @ 2012-04-01 4:52 UTC (permalink / raw)
To: daobang wang; +Cc: Mathias Burén, linux-raid
Flaky mouse button cause a premature send. Sorry for the duplicate.
On 3/31/2012 8:16 PM, daobang wang wrote:
> Thanks to Mathias and stan, Here is the detail of the configuration.
>
> 1. RAID5 with 8 2TB ST32000644NS disks , i can extend to 16 disks.
> the RAID5 created with Chunk Size of 64K and left-symmetric
>
> 2. Volume Group on RAID5 with full capacity
>
> 3. Logical Volume on the Volume Group with full capacity
LVM will create unnecessary overhead for this workload. Do not use it.
Directly format the md device with XFS and proper alignment. Again,
mkfs.xfs will do this automatically.
> 4. XFS filesystem created on the Logical Volume with option "-f -i
> size=512", and mount option is "-t xfs -o
> defaults,usrquota,grpquota,noatime,nodiratime,nobarrier,delaylog,logbsize=262144",
What kernel version are you using? logbsize=262144 has been the default
for quite some time now.
NEVER disable barriers unless you have a quality BBWC RAID controller
and a good working UPS. Are these 8 disks connected to a BBWC RAID
card? Have you verified the write back cache is working properly?
> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
> files on the XFS.
This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
It is very likely that neither 8 nor 16 disks in RAID5 will be able to
sync this rate due to excessive head seeking, as I previously mentioned.
Having LVM layered between XFS and mdraid will make this situation even
worse.
> This is the pressure testing, just verify the reliability of the
> system, we will not use it in real envrionment, 100 video streams
> writen is our goal.
So now you have an aggregate 25 MB/s random write workload with 100
writers. This is still going to likely be too much for an 8 or 16 disk
RAID5 array for the same reason as 200 threads--too many disk seeks.
> is there any clue for optimize the application?
This is the Linux-raid list. We can help you optimize your RAID, and
since many of us use XFS, we can help you there as well. I've never
seen your application so it is impossible to suggest how to optimize it.
As far as optimizing your 16 disks with mdraid and XFS, I've already
given you pointers on how you need to configure your storage for optimal
performance with this workload. However, experience tells me you simply
don't have enough spindles to sync all these parallel writers if you
also need redundancy.
The fact that you're disabling barriers likely in lieu of BBWC seems to
indicate you're not terribly worried about losing all your data in the
event of a drive failure. If this is the case, 16 x 7.2k spindles with
6 parallel writers per spindle at .25 MB/s each might work. Simply
create an md linear array with 16 disks, no LVM, format it with
$ mkfs.xfs -d agcount=48 /dev/md0
and configure your application to write each video stream file to a
different directory. Writing to multiple directories is what drives XFS
parallelism. This will allow for 96 parallel write streams, 6 to each
drive concurrently, for 6*.25 MB/s = 1.5MB/s per drive, 24 MB/s
aggregate. This might keep seeks low enough to achieve the parallel
throughput you need.
I'm guessing since you didn't comment on my previous recommendation that
you don't yet understand the benefits of linear concatenation, and using
XFS allocation groups to drive parallelism, instead of blindly assuming
striping will do the job--it often cannot. XFS on linear array can
decrease the seek rate on each drive substantially vs striped RAID.
Keeping the seek rate low increases throughput of parallel writers.
--
Stan
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: RAID5 created by 8 disks works with xfs
2012-04-01 4:52 ` Stan Hoeppner
@ 2012-04-01 8:06 ` John Robinson
0 siblings, 0 replies; 37+ messages in thread
From: John Robinson @ 2012-04-01 8:06 UTC (permalink / raw)
To: stan; +Cc: daobang wang, Mathias Burén, linux-raid
On 01/04/2012 05:52, Stan Hoeppner wrote:
[...]
>> 5. The real application is 200 D1(2Mb/s) video streams write 500MB
>> files on the XFS.
>
> This is a 50 MB/s raw stream rate with 200 writers to 200 target files.
> It is very likely that neither 8 nor 16 disks in RAID5 will be able to
> sync this rate due to excessive head seeking, as I previously mentioned.
Handwaving slightly here, but I wonder whether using a lot of write
cache will help aggregate writes here, since it's not actually random
writes. If I were to model it as 250MB written every 5 seconds from 200
writers, that might be able to be persuaded into 40 writes per second,
which shouldn't cause excessive head seeking. It would be helpful if
there were also enough additional cache that the reads for any RAID5
read-modify-write cycles mostly came from cache and didn't need seeks.
It should be possible to set this up such that, in the event of a crash
or hardware failure, the filesystem and files are consistent, though
obviously the last 5 seconds of each file is likely to be lost.
Having said that, even if it can be persuaded to work, I agree that it's
still the wrong way to go; RAID10 or indeed RAID1+linear look like much
better solutions.
Cheers,
John.
^ permalink raw reply [flat|nested] 37+ messages in thread