All of lore.kernel.org
 help / color / mirror / Atom feed
From: Olaf Hering <olaf@aepfle.de>
To: linux-kernel@vger.kernel.org
Subject: softlockup during glibc tst-robust in 2.6.18-rc3
Date: Wed, 2 Aug 2006 07:38:21 +0200	[thread overview]
Message-ID: <20060802053821.GA24356@aepfle.de> (raw)


A 'make check' in glibc mainline triggers this lockup in current Linus tree on a
4cpu p630. All I can do at this point is a reboot via sysrq b.

According to the glibc build log, tst-robust1 and tst-robust2 timed out, tst-robust3
is still running.
I'm using pseries_defconfig with HZ=100.

vmstat 9:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  0      0 3594448 649356 3226392    0    0     0  9611  366 2636 41 21 32  6  0
 2  0      0 3522548 650480 3290164    0    0     0  7151  285 3150 39 22 35  5  0
 2  2      0 3483124 650904 3305464    0    0    60  4008  197 1883 43 12 38  7  0
 5  1      0 3432532 651984 3347232    0    0    64  8288  372 1070 52  8 20 20  0
 6  0      0 3351124 657576 3415872    0    0     0  7416  318  929 47 14 20 20  0
 1  5      0 3311736 658716 3426652    0    0     0  3872  434 1541 57 18 16  8  0
 2  2      0 3295604 661336 3446192    0    0     0  3957  313  749 22 37 22 18  0
 5  0      0 3714908 661488 2996632    0    0     0  4282  428  738 53 16 25  6  0
 6  0      0 3713136 661544 3017640    0    0     0  3763  622 1422 72  9  3 15  0
 1  5      0 3621032 663432 3069864    0    0     0  8847 3173 6399 47 13 26 14  0
 7  0      0 3591824 666688 3109752    0    0     0  5911  463  789 52  9 17 22  0
 5  0      0 3467652 668708 3186068    0    0     0  4560  212  747 74  9 11  7  0
 7  0      0 3434724 669512 3205616    0    0     0  7974  368 1022 51 16 16 17  0
 0  7      0 3514044 670604 3221456    0    0     0  5403  273 1166 66 12  7 15  0
 3  1      0 3926980 671000 2752576    0    0     0  4643  306 1226 45 28 16 11  0
 5  1      0 3907264 671708 2794516    0    0     0  6526 1024 2015 67  9  9 16  0
 6  0      0 3892040 673212 2842900    0    0     0  5431  491 1447 64 13  8 15  0
 9  0      0 3882072 674228 2880744    0    0     0  8311  377  948 64  9 14 14  0
 5  2      0 3828580 678084 2936000    0    0     0  6148  416  722 47 13 19 21  0
 3  2      0 3764872 679672 2964268    0    0     0  5866  386  909 49 16 12 23  0
 3  1      0 3705840 683140 3029056    0    0    34  8627  304  640 47 10 26 17  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  1      0 3665020 684576 3064740    0    0     0  7870  275 1132 61 13 16 10  0
 6  0      0 3620108 686128 3104852    0    0     0  5588  274  848 58 13 19 10  0
 7  0      0 3556256 687700 3162420    0    0     0  8943  292  487 47  9 24 20  0
 5  0      0 3535932 689744 3225108    0    0     1  9206  359  853 62 13 10 15  0
 3  1      0 3430196 690792 3254216    0    0     0  5650  171  647 61 20 15  4  0
 1  1      0 3406220 693968 3291480    0    0     0  5216  327  390 45  9 21 25  0
 5  1      0 3390264 694292 3304472    0    0     0  4809  254  658 61  8 15 16  0
 1  2      0 3261136 699748 3378328    0    0     0  9194 1703 3241 57 13 21  9  0
 6  1      0 3224132 701144 3403520    0    0     0  5246  337  576 55  8 10 27  0
 3  1      0 3212252 703056 3446324    0    0    15  6206  240  798 73 13 10  5  0
 4  1      0 3197564 704676 3465992    0    0     0  4244  339  585 65  8  8 18  0
 5  3      0 3150860 706208 3543040    0    0     0  9143  320  800 65 13 11 11  0
 6  0      0 3100700 706840 3562112    0    0     0  3948  312  923 69 14  6 10  0
 6  0      0 3043272 707752 3625620    0    0     0  8512  330  676 67 10  7 16  0
 5  0      0 3020464 708184 3648092    0    0     0  4369  331 1275 64 11  8 17  0
 7  0      0 2901664 711148 3692596    0    0     0  6477  288 1020 78 13  5  4  0
 7  0      0 2909912 712204 3738048    0    0     0  3596  320  514 80  6  2 13  0
 6  0      0 2823392 712292 3773040    0    0     1  8041  234  556 79 16  2  2  0
 5  0      0 2813468 712476 3788940    0    0     0  2408  215  408 79  5  5 11  0
 6  0      0 2823888 712580 3801740    0    0     0  2435 2577 2917 81  8  4  7  0
 6  2      0 2815312 712604 3805240    0    0     0  2678  188  456 71 16  8  4  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  0      0 2803880 712628 3809752    0    0     0  1141  168  412 87  5  1  7  0
 5  0      0 2747044 713268 3844956    0    0     0  5545 1660 3907 72 10  7 11  0
 7  0      0 2724024 713308 3845820    0    0     0  1165  294 1228 77 12  7  4  0
 7  0      0 2753292 713356 3846460    0    0     0   936  242  909 85  9  3  4  0
 2  4      0 2781840 713392 3848944    0    0     0  1380  351 1188 73 12  6 10  0
 6  0      0 2764904 713424 3849464    0    0     0   406  254 1160 81 11  5  3  0
 6  0      0 2750232 713464 3850372    0    0     0  1400  337 1060 77 13  4  6  0
 4  0      0 3242124 713464 3375868    0    0     0  1352  304  743 68 18  7  6  0
 5  0      0 3210540 713484 3379908    0    0     0  1621  713 1667 77 11  3 10  0
 5  0      0 3230844 713552 3400784    0    0     0   721  379 1874 79 16  5  0  0
 7  0      0 3170040 714612 3425764    0    0     0  5652  532 1541 57 13 13 17  0
 8  0      0 3177324 715520 3451264    0    0     0  4188  477 1852 65 22 11  2  0
 6  0      0 3154648 715592 3461440    0    0    12  4090  370 2125 70 11  7 12  0
 7  2      0 3155840 715628 3453288    0    0     0  4634  283  832 77 11  7  5  0
 5  0      0 3063224 716436 3497000    0    0     0  5470  661 2002 74 14  7  5  0
 2  1      0 3328280 716524 3311632    0    0     0  3656  626 1604 61 15  8 17  0
 1  1      0 3592780 716576 3013692    0    0     0  4708  559 1442 64 20 12  5  0
 7  0      0 3530808 718712 3039400    0    0     0  4010  575 1256 61 11 14 14  0
 8  0      0 3454940 720548 3067656    0    0     0  3148  545 1317 59 14 16 11  0
 6  0      0 3529868 721480 3107216    0    0     0  4972  354 1233 47 11 30 12  0
15  0      0 3495260 722744 3136836    0    0     2  6173  515 1734 56 21  8 15  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  1      0 3454148 723992 3160840    0    0     5  4756  347 1590 47 19 15 19  0
 6  1      0 3414760 724996 3222176    0    0     0  8916  354 1350 65 15 13  7  0
 4  1      0 3378252 725508 3265320    0    0     0  7074  492 3057 56 14 14 16  0
 4  0      0 3353492 726692 3291232    0    0     0  4771  341 1657 62 19 14  5  0
 2  1      0 3325640 726976 3307088    0    0     0  5842  400 1106 51 11 16 22  0
 5  0      0 3221244 730284 3353696    0    0    63  5809  381  786 58 13 10 19  0
 5  0      0 3241588 731768 3389132    0    0     0  5178  352  898 59 11  9 20  0
 3  1      0 3139184 734544 3430620    0    0     0  4730  260 1592 64 17 17  3  0
 5  0      0 3087028 735380 3459036    0    0     0  4708  350  926 51 12 20 17  0
 1  1      0 3100012 737632 3506632    0    0     0  6953  327 1702 55 20 18  7  0
 1  1      0 2993264 738864 3572892    0    0     0  7829  242 1429 67 15 12  6  0
 1  1      0 2893440 740268 3635704    0    0    12  8215  391  998 49 12 22 17  0
 3  3      0 2818100 741556 3670112    0    0   272  4958  361 1450 70 16 10  3  0
 5  0      0 2782916 742268 3682148    0    0     0  2568  379 1513 54 25 12  9  0
 5  1      0 2900024 742472 3689300    0    0     0  3663  338  785 57 10 10 23  0
 7  0      0 2895124 744224 3730860    0    0     0  4342 1441 2959 63 16 12  9  0
 5  0      0 2853072 745136 3745572    0    0     0  5545  580 1108 42 10 26 22  0
 4  0      0 2800308 746304 3758752    0    0     0  2893  367 1063 53 11 13 23  0
 6  0      0 2762444 746560 3760404    0    0     0  2794  390 1062 70 11  8 11  0
 7  1      0 3288240 746768 3348092    0    0     9  4522  476 1577 63 28  8  1  0
 6  0      0 3227632 746996 3372568    0    0     0  6144  626 2074 51 16 17 16  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  2      0 3164676 747660 3403032    0    0     0  4810  514 1313 58 13 10 20  0
 1  5      0 3161416 748372 3434648    0    0     0  4931  470 1948 51 17 21 11  0
 5  0      0 3145284 748500 3445004    0    0     0  2307  382 4734 35 16 22 27  0
 1  4      0 3134408 748528 3450084    0    0     0  3536  212 6309 52 26 16  7  0
 4  2      0 3142704 748568 3466876    0    0     0  3669  342 4596 58 13 11 18  0
 4  0      0 3087936 748796 3496352    0    0     0  5476 1699 7018 35 13 34 18  0
 1  5      0 3012732 749608 3511964    0    0     0  4274  302 6475 54 17 20  9  0
 0  2      0 3031792 752004 3552032    0    0     0  4940  358 4226 42 15 28 16  0
 3  2      0 3009668 752568 3560248    0    0     0  2615  419 1694 48 15 17 20  0
 0  1      0 2979776 755504 3593816    0    0     0  4196  418 1940 59 17 13 11  0
 5  0      0 3106656 755744 3467372    0    0     0  2595  398 1833 44 20 18 18  0
 1  5      0 3445968 756136 3147784    0    0     0  2899  377 2060 40 25 26  9  0
 1  1      0 3387900 756996 3215964    0    0     0  3921 1496 3881 50 20 20  9  0
 0  5      0 3317200 757236 3259132    0    0     0  9359 1984 3312 24  6 50 21  0
 6  0      0 3287500 757516 3273288    0    0     0  2831  500 1302 48  9 24 19  0
 4  1      0 3317400 758532 3295300    0    0     0  3979  280 1528 37 11 31 21  0
11  3      0 3284816 758776 3310512    0    0     0  4748 6138 57699 51 21 18 10  0
20  0      0 3215328 759388 3335228    0    0     0  2984 16304 146397 52 31 12  5  0
10  0      0 3137744 761108 3364864    0    0     0  3570 14833 100599 54 29 11  6  0
 3  4      0 3185936 762204 3383328    0    0     0  4665 5853 39923 61 16 17  6  0
 3  1      0 3197960 763204 3397152    0    0     0  4304 7602 41247 61 17 18  4  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  1      0 3156060 764436 3447752    0    0     0  6897 8428 55444 63 19  9 10  0
 8  0      0 3056252 764796 3502828    0    0     0  8040 9656 66314 67 20  8  5  0
 3  6      0 3037412 767668 3565072    0    0     0  8298 4995 45531 57 20  8 15  0
10  0      0 2932320 769380 3583160    0    0     0  3193  371 1008 34 35 22  9  0
BUG: soft lockup detected on CPU#1!
Call Trace:
[C000000087713610] [C00000000000EBDC] .show_stack+0x68/0x1b0 (unreliable)
[C0000000877136B0] [C000000000081230] .softlockup_tick+0xf0/0x128
[C000000087713760] [C00000000005961C] .run_local_timers+0x1c/0x30
[C0000000877137E0] [C0000000000202BC] .timer_interrupt+0xa8/0x474
[C0000000877138C0] [C0000000000034DC] decrementer_common+0xdc/0x100
--- Exception: 901 at .handle_futex_death+0x28/0x98
    LR = .compat_exit_robust_list+0x108/0x1ac
[C000000087713BB0] [C0000000FAB61810] 0xc0000000fab61810 (unreliable)
[C000000087713C30] [C00000000006DA60] .compat_exit_robust_list+0x108/0x1ac
[C000000087713D00] [C000000000050C6C] .do_exit+0x1f8/0x980
[C000000087713DB0] [C0000000000514C0] .complete_and_exit+0x0/0x2c
[C000000087713E30] [C00000000000861C] syscall_exit+0x0/0x40
 9  2      0 2939520 771364 3602556    0    0     0  3320  267  478 30 39 24  7  0
 7  1      0 2865436 771984 3619916    0    0     0  2995  198  536 32 40 23  5  0
12  0      0 2841640 772456 3634672    0    0     0  2596  193  447 36 36 22  6  0
BUG: soft lockup detected on CPU#3!
Call Trace:
[C000000131003610] [C00000000000EBDC] .show_stack+0x68/0x1b0 (unreliable)
[C0000001310036B0] [C000000000081230] .softlockup_tick+0xf0/0x128
[C000000131003760] [C00000000005961C] .run_local_timers+0x1c/0x30
[C0000001310037E0] [C0000000000202BC] .timer_interrupt+0xa8/0x474
[C0000001310038C0] [C0000000000034DC] decrementer_common+0xdc/0x100
--- Exception: 901 at .handle_futex_death+0x30/0x98
    LR = .compat_exit_robust_list+0x108/0x1ac
[C000000131003BB0] [C000000136FBF810] 0xc000000136fbf810 (unreliable)
[C000000131003C30] [C00000000006DA60] .compat_exit_robust_list+0x108/0x1ac
[C000000131003D00] [C000000000050C6C] .do_exit+0x1f8/0x980
[C000000131003DB0] [C0000000000514C0] .complete_and_exit+0x0/0x2c
[C000000131003E30] [C00000000000861C] syscall_exit+0x0/0x40
10  0      0 2891040 772456 3630992    0    0     0   777   64  177 21 55 24  0  0
10  0      0 2891940 772456 3630792    0    0     0   177   33  172  0 50 50  0  0
10  0      0 2892560 772456 3630792    0    0     0   160   28  168  0 50 50  0  0
25  0      0 2891464 772456 3630864    0    0     0   164   53  103  1 64 34  0  0
BUG: soft lockup detected on CPU#0!
Call Trace:
[C0000000A4307610] [C00000000000EBDC] .show_stack+0x68/0x1b0 (unreliable)
[C0000000A43076B0] [C000000000081230] .softlockup_tick+0xf0/0x128
[C0000000A4307760] [C00000000005961C] .run_local_timers+0x1c/0x30
[C0000000A43077E0] [C0000000000202BC] .timer_interrupt+0xa8/0x474
[C0000000A43078C0] [C0000000000034DC] decrementer_common+0xdc/0x100
--- Exception: 901 at .handle_futex_death+0x5c/0x98
    LR = .compat_exit_robust_list+0x108/0x1ac
[C0000000A4307BB0] [C0000000D62E6810] 0xc0000000d62e6810 (unreliable)
[C0000000A4307C30] [C00000000006DA60] .compat_exit_robust_list+0x108/0x1ac
[C0000000A4307D00] [C000000000050C6C] .do_exit+0x1f8/0x980
[C0000000A4307DB0] [C0000000000514C0] .complete_and_exit+0x0/0x2c
[C0000000A4307E30] [C00000000000861C] syscall_exit+0x0/0x40
25  0      0 2891464 772456 3630868    0    0     0   101   16   30  0 77 23  0  0
26  0      0 2891472 772456 3630868    0    0     0     0   14   27  0 75 25  0  0
28  0      0 2891440 772456 3630868    0    0     0     0   16   28  0 75 25  0  0
28  0      0 2891440 772456 3630868    0    0     0     5   14   29  0 75 25  0  0
29  0      0 2891424 772456 3630868    0    0     0     0   20   28  0 75 25  0  0
29  0      0 2891388 772456 3630868    0    0     0    20   20   35  0 75 25  0  0
29  0      0 2891264 772456 3630868    0    0     0    10   32   30  0 75 25  0  0
29  0      0 2891212 772456 3630868    0    0     0    20   18   31  0 75 25  0  0
29  0      0 2890544 772456 3630868    0    0     0    41   22   39  0 75 25  0  0
29  0      0 2890664 772456 3630868    0    0     0     0   14   26  0 75 25  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
29  0      0 2890632 772456 3630868    0    0     0     0   13   29  0 75 25  0  0
29  0      0 2890616 772456 3630868    0    0     0     0   14   28  0 75 25  0  0
29  0      0 2890616 772456 3630868    0    0     0     0   23   28  0 75 25  0  0
29  0      0 2890616 772456 3630868    0    0     0     0   11   27  0 75 25  0  0
29  0      0 2890580 772456 3630868    0    0     0    17   15   29  0 75 25  0  0
29  0      0 2889748 772456 3630868    0    0     0    42   23   42  0 75 25  0  0
29  0      0 2889984 772456 3630868    0    0     0     0   15   32  0 75 25  0  0
29  0      0 2889968 772456 3630868    0    0     0     0   15   32  0 75 25  0  0
29  0      0 2889968 772456 3630868    0    0     0     0   15   32  0 75 25  0  0
29  0      0 2890060 772456 3630868    0    0     0     0   22   31  0 75 25  0  0
29  0      0 2890060 772456 3630868    0    0     0     0   11   28  0 75 25  0  0
29  0      0 2890060 772456 3630868    0    0     0     0   14   27  0 75 25  0  0
29  0      0 2890060 772456 3630868    0    0     0     0   17   27  0 75 25  0  0
29  0      0 2890060 772456 3630868    0    0     0     0   11   28  0 75 25  0  0
29  0      0 2890044 772456 3630868    0    0     0     0   14   58  0 75 25  0  0
29  0      0 2890044 772456 3630868    0    0     0     2   13  107  0 75 25  0  0
29  0      0 2890044 772456 3630868    0    0     0     0   24  105  0 75 25  0  0
29  0      0 2890012 772456 3630868    0    0     0     0   13  107  0 75 25  0  0
29  0      0 2890012 772456 3630868    0    0     0     0   14  105  0 75 25  0  0
29  0      0 2890012 772456 3630868    0    0     0     0   19  105  0 75 25  0  0
29  0      0 2890012 772456 3630868    0    0     0     0   14  105  0 75 25  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
29  0      0 2889996 772456 3630868    0    0     0     0   14  106  0 75 25  0  0
29  0      0 2889996 772456 3630868    0    0     0     0   26  106  0 75 25  0  0
29  0      0 2889996 772456 3630868    0    0     0     0   12  105  0 75 25  0  0
31  0      0 2789336 772560 3633644    0    0     0     2   17  107  0 75 25  0  0
31  0      0 2789336 772560 3633644    0    0     0     0   17  105  0 75 25  0  0
31  0      0 2789336 772560 3633644    0    0     0     0   16  106  0 75 25  0  0
31  0      0 2789332 772560 3633644    0    0     0    12   13  105  0 75 25  0  0
31  0      0 2789316 772560 3633644    0    0     0     0   22  107  0 75 25  0  0
31  0      0 2789316 772560 3633644    0    0     0     0   16  105  0 75 25  0  0
31  0      0 2789284 772560 3633644    0    0     0     0   16  107  0 75 25  0  0
31  0      0 2789284 772560 3633644    0    0     0     0   15  106  0 75 25  0  0
31  0      0 2789284 772560 3633644    0    0     0     0   16  105  0 75 25  0  0
31  0      0 2789284 772560 3633644    0    0     0     0   15  105  0 75 25  0  0
31  0      0 2789284 772560 3633644    0    0     0     2   13  107  0 75 25  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
31  0      0 2789268 772560 3633644    0    0     0     0   25  110  0 75 25  0  0
31  0      0 2789268 772560 3633644    0    0     0     0   14  108  0 75 25  0  0
31  0      0 2789268 772560 3633644    0    0     0     0   15  109  0 75 25  0  0
31  0      0 2789236 772560 3633644    0    0     0     0   18  110  0 75 25  0  0
31  0      0 2789236 772560 3633644    0    0     0     0   13  105  0 75 25  0  0
31  0      0 2789236 772560 3633644    0    0     0     0   16  105  0 75 25  0  0
31  0      0 2789236 772560 3633644    0    0     0     0   16  105  0 75 25  0  0
31  0      0 2789220 772560 3633644    0    0     0     0   29  107  0 75 25  0  0
31  0      0 2789220 772560 3633644    0    0     0     0   16  104  0 75 25  0  0
31  0      0 2789220 772560 3633644    0    0     0     2   20  107  0 75 25  0  0
31  0      0 2789220 772560 3633644    0    0     0     0   19  105  0 75 25  0  0
31  0      0 2789204 772560 3633644    0    0     0     0   11  106  0 75 25  0  0
31  0      0 2789188 772560 3633644    0    0     0     0   16  105  0 75 25  0  0
31  0      0 2789172 772560 3633644    0    0     0     0   22  106  0 75 25  0  0
31  0      0 2788772 772560 3633644    0    0     0    33   25  116  0 75 25  0  0
31  0      0 2788756 772560 3633644    0    0     0     6   18  107  0 75 25  0  0
31  0      0 2788756 772560 3633644    0    0     0     0   15  106  0 75 25  0  0
31  0      0 2788756 772560 3633644    0    0     0     0   17  105  0 75 25  0  0
31  0      0 2788756 772560 3633644    0    0     0     0   12  105  0 75 25  0  0
31  0      0 2788756 772560 3633644    0    0     0     2   13  107  0 75 25  0  0
31  0      0 2788708 772560 3633644    0    0     0     0   27  111  0 75 25  0  0
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 

             reply	other threads:[~2006-08-02  5:38 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2006-08-02  5:38 Olaf Hering [this message]
2006-08-03 19:23 ` softlockup during glibc tst-robust in 2.6.18-rc3 Olaf Hering
2006-08-04 19:35   ` [PATCH] futex: Apply recent futex fixes to futex_compat Thomas Gleixner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20060802053821.GA24356@aepfle.de \
    --to=olaf@aepfle.de \
    --cc=linux-kernel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.