public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: William Lee Irwin III <wli@holomorphy.com>
To: Andrew Morton <akpm@digeo.com>,
	bzzz@tmi.comex.ru, adilger@clusterfs.com,
	linux-kernel@vger.kernel.org, ext2-devel@lists.sourceforge.net
Subject: Re: [PATCH] concurrent block allocation for ext2 against 2.5.64
Date: Sat, 15 Mar 2003 00:24:31 -0800	[thread overview]
Message-ID: <20030315082431.GG5891@holomorphy.com> (raw)
In-Reply-To: <20030315070511.GQ20188@holomorphy.com>

On Fri, Mar 14, 2003 at 10:44:13PM -0800, Andrew Morton wrote:
>> Nope.  What we're trying to measure here is pure in-memory lock contention,
>> locked bus traffic, context switches, etc, etc.  To do that we need to get
>> the IO system out of the picture.
>> One way to do that is to increase /proc/sys/vm/dirty_ratio and
>> dirty_background_ratio to 70% or so.  You can still hit IO wait if someone
>> tries to truncate a file which pdflush is writing out, so increase
>> dirty_expire_centisecs and dirty_writeback_centisecs to 1000000000 or so...
>> Then, on the second run, when all the required metadata blocks are in
>> pagecache you should be able to get an IO-free run.
> 
On Fri, Mar 14, 2003 at 11:05:11PM -0800, William Lee Irwin III wrote:
> Oh, sorry, I did increase dirty_ratio and dirty_background_ratio to 99,
> I forgot about dirty_writeback_centisecs though, I'll re-run with that.

Next pass involves lockmeter:

$ cat /proc/sys/vm/dirty_expire_centisecs
360000
$ cd /test/wli
$ ls
$ (time dbench 128) |& tee -a ~/dbench.output.log.7
zsh: correct '~/dbench.output.log.7' to '~/dbench.output.log.6' [nyae]? n
128 clients started
   0     62477  206.65 MB/sec
Throughput 206.651 MB/sec 128 procs
dbench 128  143.50s user 3258.66s system 1574% cpu 3:36.04 total

vma      samples  %-age       symbol name
c0106ff4 7617343  29.5286     default_idle
c01dc3b0 5212934  20.2079     __copy_to_user_ll
c01dc418 1806434  7.00263     __copy_from_user_ll
c0264bd0 1595815  6.18617     sync_buffer
c0108140 712115   2.76051     .text.lock.semaphore
c0119058 621494   2.40922     try_to_wake_up
c011a1bc 409622   1.5879      schedule
c0107d0c 278704   1.08039     __down
c011c4ff 263802   1.02263     .text.lock.sched
c0152ab0 260394   1.00942     __find_get_block_slow
c011fadc 247423   0.959134    profile_hook
c0264da0 231721   0.898265    add_event_entry
c0168aac 223276   0.865528    d_lookup
c01dc510 212968   0.825569    atomic_dec_and_lock
c0119dac 208443   0.808028    scheduler_tick
c015f3dc 192765   0.747253    path_lookup
c01522a0 191853   0.743717    file_move
c0119860 188927   0.732375    load_balance
c0122930 168016   0.651313    current_kernel_time
c010f6b8 166633   0.645952    timer_interrupt
c013ece4 160376   0.621697    check_highmem_ptes
c0133118 155858   0.604183    find_get_page


procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo   in   cs  us sy id  wa
 0  0  48309664 568288   23360    0     0 1026    18  0  0 100  0
 0  0  48274208 568288   23392    0     0 1028   256  0  1  99  0
91  0  48192928 569056   58272    0     0 1036  1790  1 20  79  0
49  0  47769920 580224  439808    0     0 1024  5071  2 94   4  0
70  0  47437248 582720  773952    0     0 1030  2366  1 95   5  0
56  1  47140576 586656 1062272    0     0 1022  1931  1 97   3  0
86  0  46901920 581568 1307168    0     0 1025  1800  0 98   2  0
55  0  46619744 584672 1585568    0     0 1028  2584  1 93   5  0
26  1  46343712 577600 1867360    0     0 1025  3387  1 87  12  1
20  0  46033184 575936 2176384    0     0 1028  5742  2 67  30  1
27  1  45834912 577056 2366304    0     0 1027  6072  2 66  31  1
24  1  45692128 577184 2504960   16     0 1029  8056  2 56  41  1
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo   in   cs  us sy id  wa
18  0  45533280 581376 2659648   32     0 1037  8107  2 57  40  1
23  1  45386464 575264 2810592   32     0 1036  8227  2 54  42  2
 4  4  45342240 577056 2852768  360     0 1112  3217  1 14  81  5
11  1  45283040 576160 2911712  140     0 1065  9003  1 26  71  2
 8  1  45221856 580064 2968128   64     0 1043 11620  2 31  66  1
15  0  45181984 577024 3009120   92     0 1050  9451  2 40  57  1
10  0  45160224 579136 3027584   32     0 1038  9296  2 47  51  0
12  2  45198176 575200 2991840    4  1156 1205  9604  3 47  47  3
19  0  45284320 571872 2909440    0    72 1180 10532  2 42  54  1
21  0  45364064 574112 2827136    0     0 1028  9547  2 49  49  0
23  0  45368224 576384 2821504    0     0 1026  7902  3 57  41  0
22  0  45354720 571776 2839616    0   512 1127  7965  3 60  37  0
16  0  45315168 575072 2874272    0     0 1054  8367  3 59  39  0
19  0  45290592 578272 2898048   68     0 1043  8835  2 45  52  0
18  0  45244768 574432 2947424    8   640 1188  9696  2 45  51  2
18  0  45217184 576064 2973184    8     0 1028 10751  2 43  55  0
17  0  45192992 578080 2995040   24     0 1031 10860  2 40  58  0
21  0  45189408 572736 3004128    0   708 1201  8661  3 56  41  0
17  0  45219744 574112 2971392    8     0 1028  9531  3 46  51  0
26  0  45270176 575232 2919456    0     0 1028  9925  3 44  53  0
28  0  45332448 576672 2855968    0     0 1026  9073  3 50  47  0
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
17  0  45365536 578496 2821440    4     0 1028  8403  3 53  44  0
14  0  45353056 572768 2840352    0   592 1173  7842  3 60  37  0
14  0  45307808 574400 2882176    0     0 1028  8073  3 60  38  0
20  0  45287776 576320 2902208    0     0 1028  8750  3 54  43  0
17  0  45245280 578240 2943424    4     0 1025 10229  2 44  54  0
19  0  45229408 571840 2965536    0   588 1173  9809  2 44  53  0
15  1  45210848 573664 2981216   12     0 1028  9992  2 46  52  0
18  0  45189024 574880 3002016   48     0 1040  8482  2 50  48  0
 9  0  45216032 576032 2972800    0     0 1026  8795  3 53  44  0
12  0  45263968 577056 2923680    0     0 1027 10132  3 43  55  0
18  0  45312992 578080 2874432    0     0 1027  8791  3 51  46  0
20  1  45362336 575168 2828000    0   896 1118  8793  2 52  45  1
29  0  45345376 573760 2847296    0    76 1172  8315  3 56  41  0
30  0  45319840 575648 2871040    0     0 1032  7647  3 61  36  0
 6  1  45289440 576864 2898432   12     0 1026  8163  2 53  44  0
16  0  45255520 578240 2932256   12     0 1030  9372  2 44  54  0
16  0  45228000 572800 2965376    0  1016 1227 10052  2 44  52  1
15  0  45216736 573920 2974880    0     0 1080 10110  2 46  51  0
21  0  45200672 574912 2990528    0     0 1025  8989  2 54  44  0
24  0  45214304 575776 2974880    0     0 1027  9274  3 50  47  0
20  0  45263712 576608 2923520    0     0 1026  8689  3 50  48  0
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
22  0  45320800 577440 2867328    0     0 1027  9487  2 47  50  0
19  0  45344416 571488 2849888    0   632 1079  8074  3 58  39  0
23  0  45344096 573184 2848384    0     0 1132  8043  3 57  40  0
25  0  45324960 574464 2866784    0     0 1026  7457  3 64  33  0
17  0  45296992 575808 2893440    0     0 1027  9124  3 51  46  0
19  0  45265952 576992 2923648    0     0 1025  9579  2 50  48  0
23  0  45237280 577728 2949344    0     0 1030 10051  2 48  50  0
15  0  45208800 578912 2978656    0     0 1026 10072  2 47  51  0
19  0  45211552 568832 2985696    0   176 1070  8537  3 55  42  0
20  0  45215328 569792 2980736    0     0 1027  9434  3 47  50  0
12  0  45245408 570432 2949472    0     0 1026  9018  3 51  46  0
19  0  45317600 570816 2875904    0     0 1025  9767  3 47  50  0
13  0  45351712 571616 2842272    0     0 1030  9012  2 51  46  0
23  0  45342752 572640 2851232    0     0 1027  8134  3 59  38  0
23  0  45332832 573536 2859616    0     0 1026  8081  3 57  40  0
15  0  45304992 574816 2886560    0     0 1024  8929  2 54  44  0
19  0  45270752 575712 2919776    0     0 1027  9503  2 49  49  0
16  0  45230560 576896 2958208    0     0 1031 10190  2 44  54  0
22  0  45205344 577664 2981312    0     0 1024  9903  2 45  53  0
16  0  45206688 578720 2981024    4     0 1028  8860  2 53  44  0
20  0  45218080 572128 2976416    0   988 1168  9526  3 50  47  1
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
20  0  45242400 572640 2950144    0     0 1133  8446  3 52  45  0
21  0  45312608 573024 2879936    0     0 1026  9444  3 48  49  0
22  0  45347104 573664 2844576    0     0 1026  8102  3 57  40  0
15  0  45344352 574368 2847584    0     0 1027  8310  3 57  40  0
27  0  45334304 575008 2856736    0     0 1026  8763  3 52  45  0
26  0  45303328 575712 2887648    0     0 1026  9057  2 53  44  0
17  0  45269984 576704 2920704    0     0 1026  9975  2 45  52  0
20  0  45222880 578240 2966080    0     0 1027 10239  2 44  53  0
23  0  45209760 573120 2983168    4   852 1138 10067  2 45  52  1
25  0  45197792 573856 2995200    0     0 1136  8914  3 51  46  0
17  0  45216288 574208 2974976    0     0 1017  9714  2 47  51  0
15  0  45255712 574720 2935488    0     0 1026  9560  3 48  49  0
20  0  45311072 575360 2879296    0     0 1027  8887  3 55  42  0
29  0  45339360 575904 2851072    0     0 1031  7836  3 61  36  0
24  0  45341280 576480 2849056    0     0 1022  7930  3 57  40  0
27  0  45330976 577504 2858400    0     0 1025  7902  3 60  37  0
25  0  45311264 578112 2877728    0     0 1027  8213  3 56  42  0
17  0  45271712 579264 2916256   20     0 1032  9453  2 47  51  0
21  0  45236064 573632 2957344    0   864 1240 10165  2 45  52  1
21  0  45214752 574336 2977152    0     0 1026 10164  2 43  54  0
28  0  45212512 574976 2979936    0     0 1026  9132  3 52  45  0
procs -----------memory--------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
22  0  45204512 575360 2985536    0     0 1028  9087  3 51  46  0
16  0  45234976 576096 2954816    0     0 1028  9246  3 52  46  0
28  0  45284640 576608 2903360    0     0 1028  8713  3 55  43  0
30  0  45327328 577120 2860992    0     0 1026  8932  3 53  44  0
23  0  45342048 577504 2847104    0     0 1026  8421  3 55  43  0
26  0  45334656 578048 2854144    0     0 1025  7808  3 56  41  0
26  0  45315616 578720 2872352    0     0 1027  8527  3 53  45  0
18  1  45280800 574528 2911712    0   524 1028  8218  2 55  42  0
22  0  45246752 573792 2946784    0   416 1256  9411  3 45  51  1
11  0  45226464 574432 2965856    0     0 1026  9924  2 44  54  0
13  0  45214432 575296 2977056    0     0 1027  9667  2 47  51  0
24  0  45214048 575872 2976544    0     0 1026  8562  2 53  45  0
26  0  45216544 576288 2973792    0     0 1026  8803  3 48  49  0
28  0  45261472 576640 2928544    0     0 1043  9363  3 50  47  0
27  0  45304352 576928 2884896    0     0 1009  8179  3 52  45  0
27  0  45349408 577472 2839488    0     0 1028  8289  3 55  42  0
17  0  45342432 577792 2845600    0     0 1033  8466  3 56  41  0
35  0  45328672 578112 2858560    0     0 1022  8160  2 56  41  0
23  0  45290720 578464 2897344    0     0 1026  8669  2 51  47  0
17  0  45249824 579360 2938080    0     0 1027  8979  2 51  47  0
14  0  45234080 572896 2960544    0   784 1215 10500  2 43  54  1
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in   cs   us sy id  wa
20  0  45218656 573344 2974624    0     0 1030 10204  2 44  54  0
35  0  45216480 573664 2976992    0     0 1049  8737  2 51  46  0
15  0  45230048 574048 2962240    0     0 1277 10935  3 52  45  0
19  0  45256928 574432 2934816    0     0 1012  8437  3 50  47  0
18  0  45306080 574720 2885344    0     0 1025  8968  3 51  46  0
11  0  45346016 574944 2846176    0     0 1026  8805  2 53  45  0
21  0  45350688 575200 2842144    0     0 1026  7943  3 57  40  0
13  0  45315552 575520 2876352    0     0 1026  8419  2 56  42  0
24  0  45292704 575872 2897824    0     0 1028  8356  2 55  43  0
23  0  45254176 576576 2936992    0     0 1029  9594  3 49  49  0
18  0  45229856 577056 2960640    0     0 1027  9921  2 46  52  0
17  0  45209632 577600 2980864    0     0 1025  9635  2 45  53  0
26  0  45221664 577888 2967232    0     0 1045  9195  2 51  47  0
16  0  45231264 578464 2957248    0     0 1264 10989  2 51  47  0
21  0  45253088 578784 2935552    0     0 1025  8312  3 55  43  0
22  0  45310240 579040 2876992    0     0 1028  8878  3 51  47  0
23  0  45346016 579392 2841696    0     0 1026  9040  3 49  48  0
19  0  45345696 579712 2842464    0     0 1026  8184  2 58  39  0
23  0  45324000 580064 2863296    0     0 1027  8464  3 53  45  0
21  0  45285024 574240 2909024    0   748 1157  8943  2 52  45  0
17  0  45251360 574848 2941888    0     0 1081  9679  3 48  50  0
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in   cs   us sy id  wa
16  0  45233760 575296 2959616    0     0 1027 10785  2 43  55  0
21  0  45208032 575904 2983936   20     0 1034  9635  2 46  52  0
11  0  45212512 576576 2977248   12     0 1030  9593  2 48  49  0
13  0  45226848 576832 2963360    0     0 1026  9295  3 47  51  0
27  0  45264544 577088 2924704    0     0 1026  8856  3 51  46  0
17  0  45308768 577280 2880608    0     0 1027  9160  3 51  46  0
28  0  45335584 577536 2854528    0     0 1030  8855  3 52  46  0
25  0  45338336 577856 2850976    0     0 1027  7700  3 57  40  0
29  0  45326816 578048 2862496    0     0 1025  7957  3 56  42  0
33  0  45291168 578400 2897952    0     0 1028  8537  2 55  43  0
17  0  45252576 578880 2936192    0     0 1024  8634  2 53  44  0
17  0  45226912 579136 2960416    0     0 1026  9436  2 49  49  0
15  0  45206240 579392 2980928    0     0 1028  9701  2 51  47  0
26  0  45210976 579520 2978112    0     0 1026  9054  3 52  45  0
28  0  45227040 579712 2960576    0     0 1025  9191  3 47  50  0
25  0  45255328 579968 2932736    0     0 1027  8116  3 55  42  0
20  0  45301152 580096 2886624    0     0 1027  9113  3 52  45  0
23  0  45324128 572960 2870272    0   612 1129  7719  3 57  40  0
22  0  45341792 573216 2852512    0     0 1075  8123  3 54  43  0
23  0  45341728 573760 2851616    0     0 1027  8674  2 55  43  0
18  0  45306720 573984 2887360    0     0 1026  8670  2 55  43  0
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
16  0  45282016 574176 2911232    0     0 1028  9431  3 50  48  0
17  0  45243040 574432 2951456    0     0 1030  9618  3 49  48  0
21  0  45218784 574656 2974240    0     0 1025  9529  2 49  49  0
21  0  45219296 574848 2971904    0     0 1027  9826  2 48  49  0
21  0  45234464 575008 2956288    0     0 1026  9718  3 48  50  0
17  0  45265376 575168 2927712    0     0 1028  8749  3 57  40  0
22  0  45278752 575232 2914176    0     0 1026  8613  3 55  42  0
31  0  45304480 575520 2887328    0     0 1025  7851  3 60  37  0
20  0  45329888 575808 2861280    0     0 1026  8646  3 54  43  0
24  0  45331872 575936 2859552    0     0 1027  8722  2 54  44  0
21  0  45305056 576192 2886976    0     0 1026  8383  2 58  40  0
24  0  45279136 576416 2912736    0     0 1027  8586  2 57  41  0
21  0  45248736 576896 2942272    0     0 1026  9470  2 51  47  0
14  0  45223392 577184 2967328    0     0 1026 10033  2 47  50  0
19  0  45219104 577376 2971392    0     0 1026  9856  2 49  49  0
17  0  45222432 577600 2967456    0     0 1027  9484  3 47  51  0
29  0  45261472 577632 2927264    0     0 1035  8997  3 54  43  0
25  0  45271008 577792 2917664    0     0 1021  8259  3 55  42  0
24  0  45291936 578016 2897632    0     0 1027  7985  3 57  40  0
22  0  45308960 578336 2880864    0     0 1027  9218  3 52  46  0
18  0  45328160 578560 2861120    0     0 1025  8907  2 51  47  0
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in    cs  us sy id  wa
22  0  45304480 578784 2884768    0     0 1026  8257  2 52  45  0
36  0  45279776 578912 2909472    0     0 1027  8651  2 54  43  0
28  0  45254624 579104 2933280    0     0 1026  9417  3 49  48  0
22  0  45238688 579328 2949920    0     0 1031  8350  2 57  41  0
20  0  45216800 579552 2972384    0     0 1023  8923  2 51  47  0
22  0  45217504 579616 2970464    0     0 1026  9383  3 50  47  0
19  0  45249760 579808 2937920    0     0 1026  9535  2 47  50  0
21  1  45283680 580480 2902336   24     0 1032  9435  3 49  49  0
26  0  45316960 575424 2873696   52   800 1237  8920  3 57  37  4
18  2  45365152 579776 2822816  220     0 1079  8644  3 56  40  1
21 10  45406880 576064 2783904  216  1104 1219  9441  3 59  30  8
19 13  45499040 576800 2691584  184   252 1229 14054  2 52  31 15
22 26  45560736 578432 2627872  544   584 1204 11215  2 63  19 16
36 49  45656672 585472 2524128  616   288 1236 21734  2 61  14 23
34 47  45793248 580544 2391296  564   384 1251 26024  2 67  10 21
20  5  45992672 582432 2189312  272   896 1338 19731  2 75   9 15
16 107 46253536 578656 1932672  832   524 1157 41761  1 54   9 36
 1 106 46416608 575776 1792800  472  1200 1212 37282  0 23  11 66
 7 100 46468000 578784 1739456  392  1776 1201 20277  0  7   4 89
 1 100 46538208 571104 1679872  524    16 1234 22199  0  8   3 88
 0 99  46624480 578016 1587840  892     0 1255 32708  0 11   3 85
procs -----------memory---------- -----io---- --system-- ----cpu----
 r  b   free     buff   cache    bi    bo  in   cs   us sy  id wa
16 95  46728928 573920 1487168  916    44 1259 35577  0 12   2 86
 1 98  46829792 570080 1392160  860    52 1261 32797  0 12   4 85
10 94  46924640 576640 1292160  860     0 1251 33826  0 12   1 87
 8 91  47017376 572576 1205824  808    24 1245 29650  0 10   2 87
 2 90  47122592 579392 1096864  904     0 1256 35883  0 13   2 86
11 85  47241056 578784  978976  844    24 1247 36740  0 13   4 83
 8 81  47411872 578720  815584  892     4 1260 46037  0 18   5 77
 5 66  47588576 576832  651200  740     4 1229 42522  0 17   9 74
 8 48  47851872 573696  413280  760     4 1237 43504  0 19  15 66
 5 24  48114848 579616  164448  760     0 1244 31623  0 15  25 60
 0  0  48296032 575456   21280  396     0 1155  7253  0  6  70 24
 0  0  48296160 575456   21408    4     0 1028    88  0  1  99  0
 0  0  48295456 575488   21664   44     0 1037   174  0  1  98  0
 0  0  48295456 575488   21664    0     0 1025    20  0  0 100  0
 0  0  48295456 575488   21664    0     0 1025    18  0  0 100  0
 0  0  48295456 575488   21664    0     0 1025    22  0  0 100  0

  reply	other threads:[~2003-03-15  8:14 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2003-03-13  8:55 [PATCH] concurrent block allocation for ext2 against 2.5.64 Alex Tomas
2003-03-13  9:58 ` Andrew Morton
2003-03-13 19:17   ` Alex Tomas
2003-03-13 22:25     ` Andrew Morton
2003-03-13 23:03       ` Andreas Dilger
2003-03-13 23:10         ` Andrew Morton
2003-03-13 23:03       ` Alex Tomas
2003-03-13 23:25         ` Andrew Morton
2003-03-13 23:56     ` Andreas Dilger
2003-03-14  7:20       ` Alex Tomas
2003-03-14 20:59         ` Andreas Dilger
2003-03-14 21:14           ` Alex Tomas
2003-03-15  4:37         ` William Lee Irwin III
2003-03-15  4:54           ` Andrew Morton
2003-03-15  5:30             ` William Lee Irwin III
2003-03-15  5:43               ` Martin J. Bligh
2003-03-15  5:50                 ` William Lee Irwin III
2003-03-15  5:49             ` William Lee Irwin III
2003-03-15  6:20               ` William Lee Irwin III
2003-03-15  6:44                 ` Andrew Morton
2003-03-15  7:05                   ` William Lee Irwin III
2003-03-15  8:24                     ` William Lee Irwin III [this message]
2003-03-15  9:47                       ` William Lee Irwin III
2003-03-15 11:58                         ` William Lee Irwin III
2003-03-15 12:08                           ` Andrew Morton
2003-03-15 12:25                             ` William Lee Irwin III
2003-03-15  8:16                   ` [Ext2-devel] " Alex Tomas
2003-03-15  8:29                     ` William Lee Irwin III
2003-03-15  8:32                       ` Alex Tomas
2003-03-15  9:23                         ` William Lee Irwin III
2003-03-14 18:25       ` Martin J. Bligh
2003-03-14 19:30       ` [Ext2-devel] " Daniel Phillips
2003-03-14 19:55         ` Andrew Morton
2003-03-13 17:39 ` [Ext2-devel] " Andreas Dilger
2003-03-13 18:43   ` Alex Tomas
2003-03-13 19:09     ` Matthew Wilcox
2003-03-13 19:39       ` Andrew Morton
2003-03-13 19:23   ` Theodore Ts'o
2003-03-13 19:44     ` Andreas Dilger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20030315082431.GG5891@holomorphy.com \
    --to=wli@holomorphy.com \
    --cc=adilger@clusterfs.com \
    --cc=akpm@digeo.com \
    --cc=bzzz@tmi.comex.ru \
    --cc=ext2-devel@lists.sourceforge.net \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox