* [RFC PATCH 0/4] xfs: parallel quota check
@ 2013-11-12 9:29 Jeff Liu
2013-11-12 10:05 ` Jeff Liu
2013-11-12 21:03 ` Dave Chinner
0 siblings, 2 replies; 6+ messages in thread
From: Jeff Liu @ 2013-11-12 9:29 UTC (permalink / raw)
To: xfs@oss.sgi.com
Hi Folks,
We have a user report about skip quota check on first mount/boot several
monthes ago, the original discussion thread can be found at:
http://oss.sgi.com/archives/xfs/2013-06/msg00170.html.
As per Dave's suggestion, it would be possible to perform quota check
in parallel, this patch series is just trying to follow up that idea.
Sorry for the too long day as I have to spent most of time dealing with
personl things in the last few monthes, I was afraid I can not quickly
follow up the review procedure. Now the nightmare is over, it's time to
revive this task.
Also, my previous test results on my laptop and a poor desktop can not
convience me that performs parallism quota check can really get benefits
compare to the current single thread as both machines are shipped with
slow disks, I even observed a little performance regression with millions
of small files(e.g, 100 bytes) as quota check is IO bound, additionaly,
it could affected by the seek time differences. Now with a Mackbook Air
I bought recently, it can show significant difference.
tests:
- create files via fs_mark (empty file/100 byte small file)
fs_mark -k -S 0 -n 100000 -D 100 -N 1000 -d /xfs -t [10|20|30|50] -s [0|100]
- mount -ouquota,pquota /dev/sdaX /storage
- run each test for 5 times and figure out the average value
test environment:
- laptop: i5-3320M CPU 4 cores, 8G ram, normal SATA disk
results of empty files via time:
- # of file(million) default patched
1 real 1m12.0661s real 1m8.328s
user 0m0.000s user 0m0.000s
sys 0m43.692s sys 0m0.048s
2 real 1m43.907s real 1m16.221s
user 0m0.004s user 0m0.000s
sys 1m32.968s sys 0m0.065s
3 real 2m36.632s real 1m48.011s
user 0m0.000s user 0m0.002s
sys 2m23.501s sys 0m0.094s
5 real 4m20.266s real 3m0.145s
user 0m0.000s user 0m0.002s
sys 3m56.264s sys 0m0.092s
results of 100 bytes files via time:
- # of file(million) default patched
1 real 1m34.492 real 1m51.268s
user 0m0.008s user 0m0.008.s
sys 0m54.432s sys 0m0.236s
3 real 3m26.687s real 3m16.152s
user 0m0.000s user 0m0.000s
sys 2m23.144s sys 0m0.088s
So with emtpy files, the performance still looks good but with small files,
this change introduced a little regression on very slow storage. I guess
this is caused by disk seek as data blocks allocated and spreads over the
disk.
In order to get some more reasonable results, I ask a friend helping
run this test on a server which were shown as following.
test environment
- 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev.
result of 100 bytes files via time:
- # of file(million) default patched
1 real 0m19.015s real 0m16.238s
user 0m0.004s user 0m0.002s
sys 0m4.358s sys 0m0.030s
2 real 0m34.106s real 0m28.300s
user 0m0.012s user 0m0.002s
sys 0m8.820s sys 0m0.035s
3 real 0m53.716s real 46.390s
user 0m0.002s user 0m0.005s
sys 0m13.396s sys 0m0.023s
5 real 2m26.361s real 2m17.415s
user 0m0.004s user 0m0.004s
sys 0m22.188s sys 0m0.023s
In this case, there is no regression although there is no noticeable
improvements. :(
test environment
- Macbook Air i7-4650U with SSD, 8G ram
- # of file(million) default patched
1 real 0m6.367s real 0m1.972s
user 0m0.008s user 0m0.000s
sys 0m2.614s sys 0m0.008s
2 real 0m3.772s real 0m15.221s
user 0m0.000s user 0m0.000s
sys 0m0.007s sys 0m6.269s
5 real 0m36.036s real 0m8.902s
user 0m0.000s user 0m0.002s
sys 0m14.025s sys 0m0.006s
Btw, The current implementation has a defeat considering the duplicated
code at [patch 0/4] xfs: implement parallism quota check at mount time.
Maybe it's better to introduce a new function xfs_bulkstat_ag() which can
be used to bulkstat inodes per ag, hence it could shared at above patch while
adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag().
As usual, critism and comments are both welcome!
Thanks,
-Jeff
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [RFC PATCH 0/4] xfs: parallel quota check 2013-11-12 9:29 [RFC PATCH 0/4] xfs: parallel quota check Jeff Liu @ 2013-11-12 10:05 ` Jeff Liu 2013-11-12 21:03 ` Dave Chinner 1 sibling, 0 replies; 6+ messages in thread From: Jeff Liu @ 2013-11-12 10:05 UTC (permalink / raw) To: xfs@oss.sgi.com Sorry, here have two typos. On 11/12, 2013 17:29 PM, Jeff Liu wrote: > Hi Folks, > > We have a user report about skip quota check on first mount/boot several > monthes ago, the original discussion thread can be found at: > http://oss.sgi.com/archives/xfs/2013-06/msg00170.html. > > As per Dave's suggestion, it would be possible to perform quota check > in parallel, this patch series is just trying to follow up that idea. > > Sorry for the too long day as I have to spent most of time dealing with > personl things in the last few monthes, I was afraid I can not quickly > follow up the review procedure. Now the nightmare is over, it's time to > revive this task. > > Also, my previous test results on my laptop and a poor desktop can not > convience me that performs parallism quota check can really get benefits > compare to the current single thread as both machines are shipped with > slow disks, I even observed a little performance regression with millions > of small files(e.g, 100 bytes) as quota check is IO bound, additionaly, > it could affected by the seek time differences. Now with a Mackbook Air > I bought recently, it can show significant difference. > > tests: > - create files via fs_mark (empty file/100 byte small file) > fs_mark -k -S 0 -n 100000 -D 100 -N 1000 -d /xfs -t [10|20|30|50] -s [0|100] > - mount -ouquota,pquota /dev/sdaX /storage > - run each test for 5 times and figure out the average value > > test environment: > - laptop: i5-3320M CPU 4 cores, 8G ram, normal SATA disk > > results of empty files via time: > - # of file(million) default patched > 1 real 1m12.0661s real 1m8.328s > user 0m0.000s user 0m0.000s > sys 0m43.692s sys 0m0.048s > > 2 real 1m43.907s real 1m16.221s > user 0m0.004s user 0m0.000s > sys 1m32.968s sys 0m0.065s > > 3 real 2m36.632s real 1m48.011s > user 0m0.000s user 0m0.002s > sys 2m23.501s sys 0m0.094s > > 5 real 4m20.266s real 3m0.145s > user 0m0.000s user 0m0.002s > sys 3m56.264s sys 0m0.092s > > results of 100 bytes files via time: > - # of file(million) default patched > 1 real 1m34.492 real 1m51.268s > user 0m0.008s user 0m0.008.s > sys 0m54.432s sys 0m0.236s > > 3 real 3m26.687s real 3m16.152s > user 0m0.000s user 0m0.000s > sys 2m23.144s sys 0m0.088s > > So with emtpy files, the performance still looks good but with small files, > this change introduced a little regression on very slow storage. I guess > this is caused by disk seek as data blocks allocated and spreads over the > disk. > > In order to get some more reasonable results, I ask a friend helping > run this test on a server which were shown as following. > > test environment > - 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev. > > result of 100 bytes files via time: > - # of file(million) default patched > 1 real 0m19.015s real 0m16.238s > user 0m0.004s user 0m0.002s > sys 0m4.358s sys 0m0.030s > > 2 real 0m34.106s real 0m28.300s > user 0m0.012s user 0m0.002s > sys 0m8.820s sys 0m0.035s > > 3 real 0m53.716s real 46.390s real 0m46.390s > user 0m0.002s user 0m0.005s > sys 0m13.396s sys 0m0.023s > > 5 real 2m26.361s real 2m17.415s > user 0m0.004s user 0m0.004s > sys 0m22.188s sys 0m0.023s > > In this case, there is no regression although there is no noticeable > improvements. :( > > test environment > - Macbook Air i7-4650U with SSD, 8G ram > > - # of file(million) default patched > 1 real 0m6.367s real 0m1.972s > user 0m0.008s user 0m0.000s > sys 0m2.614s sys 0m0.008s > > 2 real 0m3.772s real 0m15.221s real 0m15.221s real 0m3.772s > user 0m0.000s user 0m0.000s > sys 0m0.007s sys 0m6.269s > > 5 real 0m36.036s real 0m8.902s > user 0m0.000s user 0m0.002s > sys 0m14.025s sys 0m0.006s > > > Btw, The current implementation has a defeat considering the duplicated > code at [patch 0/4] xfs: implement parallism quota check at mount time. > Maybe it's better to introduce a new function xfs_bulkstat_ag() which can > be used to bulkstat inodes per ag, hence it could shared at above patch while > adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag(). > > As usual, critism and comments are both welcome! > > Thanks, > -Jeff > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/4] xfs: parallel quota check 2013-11-12 9:29 [RFC PATCH 0/4] xfs: parallel quota check Jeff Liu 2013-11-12 10:05 ` Jeff Liu @ 2013-11-12 21:03 ` Dave Chinner 2013-11-13 2:27 ` Jeff Liu 1 sibling, 1 reply; 6+ messages in thread From: Dave Chinner @ 2013-11-12 21:03 UTC (permalink / raw) To: Jeff Liu; +Cc: xfs@oss.sgi.com On Tue, Nov 12, 2013 at 05:29:15PM +0800, Jeff Liu wrote: > Hi Folks, > > We have a user report about skip quota check on first mount/boot several > monthes ago, the original discussion thread can be found at: > http://oss.sgi.com/archives/xfs/2013-06/msg00170.html. > > As per Dave's suggestion, it would be possible to perform quota check > in parallel, this patch series is just trying to follow up that idea. > > Sorry for the too long day as I have to spent most of time dealing with > personl things in the last few monthes, I was afraid I can not quickly > follow up the review procedure. Now the nightmare is over, it's time to > revive this task. > > Also, my previous test results on my laptop and a poor desktop can not > convience me that performs parallism quota check can really get benefits > compare to the current single thread as both machines are shipped with > slow disks, I even observed a little performance regression with millions > of small files(e.g, 100 bytes) as quota check is IO bound, additionaly, > it could affected by the seek time differences. Now with a Mackbook Air > I bought recently, it can show significant difference. Results look good - they definitely point out that we can improve the situation here. > In order to get some more reasonable results, I ask a friend helping > run this test on a server which were shown as following. > > test environment > - 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev. .... > > In this case, there is no regression although there is no noticeable > improvements. :( Which is no surprise - there isn't any extra IO parallelism that can be extracted from a single spindle.... > test environment > - Macbook Air i7-4650U with SSD, 8G ram > > - # of file(million) default patched > 1 real 0m6.367s real 0m1.972s > user 0m0.008s user 0m0.000s > sys 0m2.614s sys 0m0.008s > > 2 real 0m15.221s real 0m3.772s > user 0m0.000s user 0m0.000s > sys 0m6.269s sys 0m0.007s > > 5 real 0m36.036s real 0m8.902s > user 0m0.000s user 0m0.002s > sys 0m14.025s sys 0m0.006s But a SSD or large raid array does have unused IO parallelism we can exploit. ;) Note that there is also the possibility of applying too much parallelism for the underlying storage (think of a filesystem with hundreds of AGs on a limited number of spindles) and hence causing degradations due to seeking. Hence it might be worthwhile to limit the number of AGs being scanned concurrently... > Btw, The current implementation has a defeat considering the duplicated > code at [patch 0/4] xfs: implement parallism quota check at mount time. > Maybe it's better to introduce a new function xfs_bulkstat_ag() which can > be used to bulkstat inodes per ag, hence it could shared at above patch while > adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag(). Right, there are uses for AG-based parallelism of bulkstat for userspace, so exposing single AG scans via the bulkstat ioctl is something I've been intending to do for some time. Hence I'd much prefer to see xfs_bulkstat_ag() to be implemented and then the quotacheck code converted to use it rather than duplicating the algorithm and code specifically to parallelise quotacheck. I like the factoring of the bulkstat code (about time we did that), but I think the factored functions should remain in xfs-itable.c with the rest of the bulkstat code for now... Also, there's a race condition you haven't handled in the quotacheck code: xfs_qm_quotacheck_dqadjust() can now be called concurrently on a dquot from different threads to update the same dquot, and there's no locking of the dquot to prevent this. As to the workqueues for threading, it seems overly complex. You could create a permanent workqueue in xfs_init_workqueues() for this, and you can use flush_workqueue() to execute and wait for all the per-ag scans to complete once they have been queued. This gets rid of all the lists and completions from the code. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/4] xfs: parallel quota check 2013-11-12 21:03 ` Dave Chinner @ 2013-11-13 2:27 ` Jeff Liu 2013-11-13 5:40 ` Dave Chinner 0 siblings, 1 reply; 6+ messages in thread From: Jeff Liu @ 2013-11-13 2:27 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs@oss.sgi.com Thanks for your quick response! On 11/13 2013 05:03 PM, Dave Chinner wrote: > On Tue, Nov 12, 2013 at 05:29:15PM +0800, Jeff Liu wrote: >> Hi Folks, >> >> We have a user report about skip quota check on first mount/boot several >> monthes ago, the original discussion thread can be found at: >> http://oss.sgi.com/archives/xfs/2013-06/msg00170.html. >> >> As per Dave's suggestion, it would be possible to perform quota check >> in parallel, this patch series is just trying to follow up that idea. >> >> Sorry for the too long day as I have to spent most of time dealing with >> personl things in the last few monthes, I was afraid I can not quickly >> follow up the review procedure. Now the nightmare is over, it's time to >> revive this task. >> >> Also, my previous test results on my laptop and a poor desktop can not >> convience me that performs parallism quota check can really get benefits >> compare to the current single thread as both machines are shipped with >> slow disks, I even observed a little performance regression with millions >> of small files(e.g, 100 bytes) as quota check is IO bound, additionaly, >> it could affected by the seek time differences. Now with a Mackbook Air >> I bought recently, it can show significant difference. > > Results look good - they definitely point out that we can improve > the situation here. > >> In order to get some more reasonable results, I ask a friend helping >> run this test on a server which were shown as following. >> >> test environment >> - 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev. > .... >> >> In this case, there is no regression although there is no noticeable >> improvements. :( > > Which is no surprise - there isn't any extra IO parallelism that can > be extracted from a single spindle.... > >> test environment >> - Macbook Air i7-4650U with SSD, 8G ram >> >> - # of file(million) default patched >> 1 real 0m6.367s real 0m1.972s >> user 0m0.008s user 0m0.000s >> sys 0m2.614s sys 0m0.008s >> >> 2 real 0m15.221s real 0m3.772s >> user 0m0.000s user 0m0.000s >> sys 0m6.269s sys 0m0.007s >> >> 5 real 0m36.036s real 0m8.902s >> user 0m0.000s user 0m0.002s >> sys 0m14.025s sys 0m0.006s > > But a SSD or large raid array does have unused IO parallelism we can > exploit. ;) > > Note that there is also the possibility of applying too much > parallelism for the underlying storage (think of a filesystem with > hundreds of AGs on a limited number of spindles) and hence causing > degradations due to seeking. Hence it might be worthwhile to limit > the number of AGs being scanned concurrently... Ok, maybe it could be a new mount option to let user decide how to deal with it in this situation, let me think it over. > >> Btw, The current implementation has a defeat considering the duplicated >> code at [patch 0/4] xfs: implement parallism quota check at mount time. >> Maybe it's better to introduce a new function xfs_bulkstat_ag() which can >> be used to bulkstat inodes per ag, hence it could shared at above patch while >> adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag(). > > Right, there are uses for AG-based parallelism of bulkstat for > userspace, so exposing single AG scans via the bulkstat ioctl is > something I've been intending to do for some time. Hence I'd much > prefer to see xfs_bulkstat_ag() to be implemented and then the > quotacheck code converted to use it rather than duplicating the > algorithm and code specifically to parallelise quotacheck. Thanks for the confirmation, this change will be reflected in the next round of post. > > I like the factoring of the bulkstat code (about time we did that), > but I think the factored functions should remain in xfs-itable.c > with the rest of the bulkstat code for now... > > Also, there's a race condition you haven't handled in the quotacheck > code: xfs_qm_quotacheck_dqadjust() can now be called concurrently on > a dquot from different threads to update the same dquot, and there's > no locking of the dquot to prevent this. Ah, will fix it, why I have not found this problem in the previous test? :-P > > As to the workqueues for threading, it seems overly complex. You > could create a permanent workqueue in xfs_init_workqueues() for > this, and you can use flush_workqueue() to execute and wait for all > the per-ag scans to complete once they have been queued. This gets > rid of all the lists and completions from the code. At that time, I thought the workqueue should be destroyed once the quota check procedure is complete as it only run once at mount time, will take care of it. Thanks, -Jeff _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/4] xfs: parallel quota check 2013-11-13 2:27 ` Jeff Liu @ 2013-11-13 5:40 ` Dave Chinner 2013-11-13 7:32 ` Jeff Liu 0 siblings, 1 reply; 6+ messages in thread From: Dave Chinner @ 2013-11-13 5:40 UTC (permalink / raw) To: Jeff Liu; +Cc: xfs@oss.sgi.com On Wed, Nov 13, 2013 at 10:27:48AM +0800, Jeff Liu wrote: > Thanks for your quick response! > > On 11/13 2013 05:03 PM, Dave Chinner wrote: > > On Tue, Nov 12, 2013 at 05:29:15PM +0800, Jeff Liu wrote: > >> Hi Folks, > >> > >> We have a user report about skip quota check on first mount/boot several > >> monthes ago, the original discussion thread can be found at: > >> http://oss.sgi.com/archives/xfs/2013-06/msg00170.html. > >> > >> As per Dave's suggestion, it would be possible to perform quota check > >> in parallel, this patch series is just trying to follow up that idea. > >> > >> Sorry for the too long day as I have to spent most of time dealing with > >> personl things in the last few monthes, I was afraid I can not quickly > >> follow up the review procedure. Now the nightmare is over, it's time to > >> revive this task. > >> > >> Also, my previous test results on my laptop and a poor desktop can not > >> convience me that performs parallism quota check can really get benefits > >> compare to the current single thread as both machines are shipped with > >> slow disks, I even observed a little performance regression with millions > >> of small files(e.g, 100 bytes) as quota check is IO bound, additionaly, > >> it could affected by the seek time differences. Now with a Mackbook Air > >> I bought recently, it can show significant difference. > > > > Results look good - they definitely point out that we can improve > > the situation here. > > > >> In order to get some more reasonable results, I ask a friend helping > >> run this test on a server which were shown as following. > >> > >> test environment > >> - 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev. > > .... > >> > >> In this case, there is no regression although there is no noticeable > >> improvements. :( > > > > Which is no surprise - there isn't any extra IO parallelism that can > > be extracted from a single spindle.... > > > >> test environment > >> - Macbook Air i7-4650U with SSD, 8G ram > >> > >> - # of file(million) default patched > >> 1 real 0m6.367s real 0m1.972s > >> user 0m0.008s user 0m0.000s > >> sys 0m2.614s sys 0m0.008s > >> > >> 2 real 0m15.221s real 0m3.772s > >> user 0m0.000s user 0m0.000s > >> sys 0m6.269s sys 0m0.007s > >> > >> 5 real 0m36.036s real 0m8.902s > >> user 0m0.000s user 0m0.002s > >> sys 0m14.025s sys 0m0.006s > > > > But a SSD or large raid array does have unused IO parallelism we can > > exploit. ;) > > > > Note that there is also the possibility of applying too much > > parallelism for the underlying storage (think of a filesystem with > > hundreds of AGs on a limited number of spindles) and hence causing > > degradations due to seeking. Hence it might be worthwhile to limit > > the number of AGs being scanned concurrently... > Ok, maybe it could be a new mount option to let user decide how to deal > with it in this situation, let me think it over. I'd prefer that we just do it automatically. There's a diminishing return curve that adding more parallelism will result in, so as long as we don't go too far down the tail of the curve it should not be a problem. Also, keep in mind if you issue too much readahead to a block device and the queue becomes read congested, it will just drop new readahead attempts. This is another reason for limiting the parallelism and hence the amount of readahead we issue.... > >> Btw, The current implementation has a defeat considering the duplicated > >> code at [patch 0/4] xfs: implement parallism quota check at mount time. > >> Maybe it's better to introduce a new function xfs_bulkstat_ag() which can > >> be used to bulkstat inodes per ag, hence it could shared at above patch while > >> adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag(). > > > > Right, there are uses for AG-based parallelism of bulkstat for > > userspace, so exposing single AG scans via the bulkstat ioctl is > > something I've been intending to do for some time. Hence I'd much > > prefer to see xfs_bulkstat_ag() to be implemented and then the > > quotacheck code converted to use it rather than duplicating the > > algorithm and code specifically to parallelise quotacheck. > Thanks for the confirmation, this change will be reflected in the next > round of post. > > > > > I like the factoring of the bulkstat code (about time we did that), > > but I think the factored functions should remain in xfs-itable.c > > with the rest of the bulkstat code for now... > > > > Also, there's a race condition you haven't handled in the quotacheck > > code: xfs_qm_quotacheck_dqadjust() can now be called concurrently on > > a dquot from different threads to update the same dquot, and there's > > no locking of the dquot to prevent this. > Ah, will fix it, why I have not found this problem in the previous test? :-P Because it is simply assumed that the quotacheck gets the calculation correct? i.e. the calculated values are not actually validated anywhere except in xfstests that have limited scope for quotacheck parallelism... > > As to the workqueues for threading, it seems overly complex. You > > could create a permanent workqueue in xfs_init_workqueues() for > > this, and you can use flush_workqueue() to execute and wait for all > > the per-ag scans to complete once they have been queued. This gets > > rid of all the lists and completions from the code. > At that time, I thought the workqueue should be destroyed once the quota > check procedure is complete as it only run once at mount time, will take care > of it. I understand. Having a workqueue sit around idle does not take up any resources, so I don't think we need the complexity of making them dynamic... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 0/4] xfs: parallel quota check 2013-11-13 5:40 ` Dave Chinner @ 2013-11-13 7:32 ` Jeff Liu 0 siblings, 0 replies; 6+ messages in thread From: Jeff Liu @ 2013-11-13 7:32 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs@oss.sgi.com On 11/13/2013 01:40 PM, Dave Chinner wrote: > On Wed, Nov 13, 2013 at 10:27:48AM +0800, Jeff Liu wrote: >> Thanks for your quick response! >> >> On 11/13 2013 05:03 PM, Dave Chinner wrote: >>> On Tue, Nov 12, 2013 at 05:29:15PM +0800, Jeff Liu wrote: >>>> Hi Folks, >>>> >>>> We have a user report about skip quota check on first mount/boot several >>>> monthes ago, the original discussion thread can be found at: >>>> http://oss.sgi.com/archives/xfs/2013-06/msg00170.html. >>>> >>>> As per Dave's suggestion, it would be possible to perform quota check >>>> in parallel, this patch series is just trying to follow up that idea. >>>> >>>> Sorry for the too long day as I have to spent most of time dealing with >>>> personl things in the last few monthes, I was afraid I can not quickly >>>> follow up the review procedure. Now the nightmare is over, it's time to >>>> revive this task. >>>> >>>> Also, my previous test results on my laptop and a poor desktop can not >>>> convience me that performs parallism quota check can really get benefits >>>> compare to the current single thread as both machines are shipped with >>>> slow disks, I even observed a little performance regression with millions >>>> of small files(e.g, 100 bytes) as quota check is IO bound, additionaly, >>>> it could affected by the seek time differences. Now with a Mackbook Air >>>> I bought recently, it can show significant difference. >>> >>> Results look good - they definitely point out that we can improve >>> the situation here. >>> >>>> In order to get some more reasonable results, I ask a friend helping >>>> run this test on a server which were shown as following. >>>> >>>> test environment >>>> - 16core, 25G ram, normal SATA disk, but the XFS is resides on a loop dev. >>> .... >>>> >>>> In this case, there is no regression although there is no noticeable >>>> improvements. :( >>> >>> Which is no surprise - there isn't any extra IO parallelism that can >>> be extracted from a single spindle.... >>> >>>> test environment >>>> - Macbook Air i7-4650U with SSD, 8G ram >>>> >>>> - # of file(million) default patched >>>> 1 real 0m6.367s real 0m1.972s >>>> user 0m0.008s user 0m0.000s >>>> sys 0m2.614s sys 0m0.008s >>>> >>>> 2 real 0m15.221s real 0m3.772s >>>> user 0m0.000s user 0m0.000s >>>> sys 0m6.269s sys 0m0.007s >>>> >>>> 5 real 0m36.036s real 0m8.902s >>>> user 0m0.000s user 0m0.002s >>>> sys 0m14.025s sys 0m0.006s >>> >>> But a SSD or large raid array does have unused IO parallelism we can >>> exploit. ;) >>> >>> Note that there is also the possibility of applying too much >>> parallelism for the underlying storage (think of a filesystem with >>> hundreds of AGs on a limited number of spindles) and hence causing >>> degradations due to seeking. Hence it might be worthwhile to limit >>> the number of AGs being scanned concurrently... >> Ok, maybe it could be a new mount option to let user decide how to deal >> with it in this situation, let me think it over. > > I'd prefer that we just do it automatically. There's a diminishing > return curve that adding more parallelism will result in, so as long > as we don't go too far down the tail of the curve it should not be a > problem. > > Also, keep in mind if you issue too much readahead to a block device > and the queue becomes read congested, it will just drop new > readahead attempts. This is another reason for limiting the > parallelism and hence the amount of readahead we issue.... Yup. Peviously, I also tried to remove the readahead mechanism to measure the results. But since those tests only run against 4 AG by default, so neither benefits nor read congested could be observed. I definitely would bear this in mind, thanks for the teaching! > >>>> Btw, The current implementation has a defeat considering the duplicated >>>> code at [patch 0/4] xfs: implement parallism quota check at mount time. >>>> Maybe it's better to introduce a new function xfs_bulkstat_ag() which can >>>> be used to bulkstat inodes per ag, hence it could shared at above patch while >>>> adjusting dquota usage per ag, i.e, xfs_qm_dqusage_adjust_perag(). >>> >>> Right, there are uses for AG-based parallelism of bulkstat for >>> userspace, so exposing single AG scans via the bulkstat ioctl is >>> something I've been intending to do for some time. Hence I'd much >>> prefer to see xfs_bulkstat_ag() to be implemented and then the >>> quotacheck code converted to use it rather than duplicating the >>> algorithm and code specifically to parallelise quotacheck. >> Thanks for the confirmation, this change will be reflected in the next >> round of post. >> >>> >>> I like the factoring of the bulkstat code (about time we did that), >>> but I think the factored functions should remain in xfs-itable.c >>> with the rest of the bulkstat code for now... >>> >>> Also, there's a race condition you haven't handled in the quotacheck >>> code: xfs_qm_quotacheck_dqadjust() can now be called concurrently on >>> a dquot from different threads to update the same dquot, and there's >>> no locking of the dquot to prevent this. >> Ah, will fix it, why I have not found this problem in the previous test? :-P > > Because it is simply assumed that the quotacheck gets the > calculation correct? i.e. the calculated values are not actually > validated anywhere except in xfstests that have limited scope for > quotacheck parallelism... Yup, I only verified that results via xfs_quota -xc 'report -[i|h]' before. > >>> As to the workqueues for threading, it seems overly complex. You >>> could create a permanent workqueue in xfs_init_workqueues() for >>> this, and you can use flush_workqueue() to execute and wait for all >>> the per-ag scans to complete once they have been queued. This gets >>> rid of all the lists and completions from the code. >> At that time, I thought the workqueue should be destroyed once the quota >> check procedure is complete as it only run once at mount time, will take care >> of it. > > I understand. Having a workqueue sit around idle does not take up > any resources, so I don't think we need the complexity of making > them dynamic... Agree, that's sounds like a trade-off to me. :) Thanks, -Jeff _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-11-13 7:31 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-11-12 9:29 [RFC PATCH 0/4] xfs: parallel quota check Jeff Liu 2013-11-12 10:05 ` Jeff Liu 2013-11-12 21:03 ` Dave Chinner 2013-11-13 2:27 ` Jeff Liu 2013-11-13 5:40 ` Dave Chinner 2013-11-13 7:32 ` Jeff Liu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox