* Re: 2.6.32.3 x86_64 - XFS hangs, all I/O to D-state bug (again)
2010-02-11 14:13 ` 2.6.32.3 x86_64 - XFS hangs, all I/O to D-state bug (again) Justin Piszcz
@ 2010-02-11 14:23 ` Justin Piszcz
2010-02-11 14:35 ` Justin Piszcz
0 siblings, 1 reply; 4+ messages in thread
From: Justin Piszcz @ 2010-02-11 14:23 UTC (permalink / raw)
To: linux-kernel, linux-raid, xfs, Alan Piszcz
Is this normal? After the crash I just showed, the system was rebooted,
it is resyncing now but is this normal?
$ cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[1] sda2[0]
136448 blocks [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]
129596288 blocks [2/2] [UU]
md3 : active raid5 sdj1[7] sdi1[6] sdh1[5] sdf1[3] sdg1[4] sde1[2] sdd1[1] sdc1[0]
5128001536 blocks level 5, 1024k chunk, algorithm 2 [8/8] [UUUUUUUU]
[==>..................] resync = 12.3% (90186112/732571648) finish=1046.9min speed=10226K/sec
md0 : active raid1 sdb1[1] sda1[0]
16787776 blocks [2/2] [UU]
unused devices: <none>
--
root 24 0.3 0.0 0 0 ? S 07:37 0:23 [async/mgr]
root 15514 0.2 0.0 0 0 ? S 08:04 0:11 [async/0]
root 15515 0.2 0.0 0 0 ? S 08:04 0:09 [async/1]
root 15516 0.2 0.0 0 0 ? S 08:04 0:10 [async/2]
root 15517 0.2 0.0 0 0 ? S 08:04 0:09 [async/3]
root 15518 0.2 0.0 0 0 ? S 08:04 0:10 [async/4]
root 15519 0.2 0.0 0 0 ? S 08:04 0:10 [async/5]
root 15520 0.2 0.0 0 0 ? S 08:04 0:11 [async/6]
root 15521 0.2 0.0 0 0 ? S 08:04 0:10 [async/7]
root 15522 0.2 0.0 0 0 ? S 08:04 0:10 [async/8]
root 15523 0.2 0.0 0 0 ? S 08:04 0:10 [async/9]
root 15524 0.2 0.0 0 0 ? S 08:04 0:10 [async/10]
root 15525 0.2 0.0 0 0 ? S 08:04 0:10 [async/11]
root 15526 0.2 0.0 0 0 ? S 08:04 0:11 [async/12]
root 15527 0.2 0.0 0 0 ? S 08:04 0:10 [async/13]
root 15528 0.2 0.0 0 0 ? S 08:04 0:11 [async/14]
root 15529 0.2 0.0 0 0 ? S 08:04 0:10 [async/15]
root 15530 0.2 0.0 0 0 ? S 08:04 0:10 [async/16]
root 15531 0.2 0.0 0 0 ? S 08:04 0:11 [async/17]
root 15532 0.2 0.0 0 0 ? S 08:04 0:11 [async/18]
root 15533 0.2 0.0 0 0 ? S 08:04 0:11 [async/19]
root 15534 0.2 0.0 0 0 ? S 08:04 0:11 [async/20]
root 15535 0.2 0.0 0 0 ? S 08:04 0:11 [async/21]
root 15536 0.2 0.0 0 0 ? S 08:04 0:10 [async/22]
root 15537 0.2 0.0 0 0 ? S 08:04 0:10 [async/23]
root 15538 0.2 0.0 0 0 ? S 08:04 0:11 [async/24]
root 15539 0.2 0.0 0 0 ? S 08:04 0:11 [async/25]
root 15540 0.2 0.0 0 0 ? S 08:04 0:11 [async/26]
root 15541 0.2 0.0 0 0 ? S 08:04 0:11 [async/27]
root 15542 0.2 0.0 0 0 ? S 08:04 0:10 [async/28]
root 15543 0.2 0.0 0 0 ? S 08:04 0:12 [async/29]
root 15544 0.2 0.0 0 0 ? S 08:04 0:10 [async/30]
root 15545 0.2 0.0 0 0 ? S 08:04 0:11 [async/31]
root 15546 0.2 0.0 0 0 ? R 08:04 0:11 [async/32]
root 15547 0.2 0.0 0 0 ? S 08:04 0:11 [async/33]
root 15548 0.2 0.0 0 0 ? S 08:04 0:11 [async/34]
root 15549 0.2 0.0 0 0 ? R 08:04 0:11 [async/35]
root 15550 0.2 0.0 0 0 ? S 08:04 0:11 [async/36]
root 15551 0.2 0.0 0 0 ? S 08:04 0:11 [async/37]
root 15552 0.2 0.0 0 0 ? S 08:04 0:11 [async/38]
root 15553 0.2 0.0 0 0 ? S 08:04 0:10 [async/39]
root 15554 0.2 0.0 0 0 ? S 08:04 0:11 [async/40]
root 15555 0.2 0.0 0 0 ? S 08:04 0:11 [async/41]
root 15556 0.2 0.0 0 0 ? S 08:04 0:11 [async/42]
root 15557 0.2 0.0 0 0 ? S 08:04 0:10 [async/43]
root 15558 0.2 0.0 0 0 ? S 08:04 0:10 [async/44]
root 15559 0.2 0.0 0 0 ? R 08:04 0:11 [async/45]
root 15560 0.2 0.0 0 0 ? S 08:04 0:10 [async/46]
root 15561 0.2 0.0 0 0 ? S 08:04 0:11 [async/47]
root 15562 0.2 0.0 0 0 ? S 08:04 0:10 [async/48]
root 15563 0.2 0.0 0 0 ? S 08:04 0:11 [async/49]
root 15564 0.2 0.0 0 0 ? S 08:04 0:11 [async/50]
root 15565 0.2 0.0 0 0 ? R 08:04 0:11 [async/51]
root 15566 0.2 0.0 0 0 ? S 08:04 0:11 [async/52]
root 15567 0.2 0.0 0 0 ? S 08:04 0:11 [async/53]
root 15568 0.2 0.0 0 0 ? S 08:04 0:11 [async/54]
root 15569 0.2 0.0 0 0 ? S 08:04 0:10 [async/55]
root 15570 0.2 0.0 0 0 ? S 08:04 0:11 [async/56]
root 15571 0.2 0.0 0 0 ? S 08:04 0:10 [async/57]
root 15572 0.2 0.0 0 0 ? S 08:04 0:12 [async/58]
root 15573 0.2 0.0 0 0 ? S 08:04 0:10 [async/59]
root 15574 0.2 0.0 0 0 ? S 08:04 0:12 [async/60]
root 15575 0.2 0.0 0 0 ? S 08:04 0:11 [async/61]
root 15576 0.2 0.0 0 0 ? S 08:04 0:10 [async/62]
root 15577 0.2 0.0 0 0 ? R 08:04 0:12 [async/63]
root 15578 0.2 0.0 0 0 ? S 08:04 0:11 [async/64]
root 15579 0.2 0.0 0 0 ? S 08:04 0:11 [async/65]
root 15580 0.2 0.0 0 0 ? S 08:04 0:11 [async/66]
root 15581 0.2 0.0 0 0 ? S 08:04 0:10 [async/67]
root 15582 0.2 0.0 0 0 ? S 08:04 0:12 [async/68]
root 15583 0.2 0.0 0 0 ? S 08:04 0:10 [async/69]
root 15584 0.2 0.0 0 0 ? R 08:04 0:11 [async/70]
root 15585 0.2 0.0 0 0 ? S 08:04 0:10 [async/71]
root 15586 0.2 0.0 0 0 ? S 08:04 0:11 [async/72]
root 15587 0.2 0.0 0 0 ? S 08:04 0:11 [async/73]
root 15588 0.2 0.0 0 0 ? S 08:04 0:11 [async/74]
root 15589 0.2 0.0 0 0 ? S 08:04 0:11 [async/75]
root 15590 0.2 0.0 0 0 ? S 08:04 0:11 [async/76]
root 15591 0.2 0.0 0 0 ? S 08:04 0:11 [async/77]
root 15592 0.2 0.0 0 0 ? S 08:04 0:11 [async/78]
root 15593 0.2 0.0 0 0 ? S 08:04 0:11 [async/79]
root 15594 0.2 0.0 0 0 ? S 08:04 0:11 [async/80]
root 15595 0.2 0.0 0 0 ? S 08:04 0:10 [async/81]
root 15596 0.2 0.0 0 0 ? S 08:04 0:11 [async/82]
root 15597 0.2 0.0 0 0 ? S 08:04 0:12 [async/83]
root 15598 0.2 0.0 0 0 ? S 08:04 0:11 [async/84]
root 15599 0.2 0.0 0 0 ? S 08:04 0:11 [async/85]
root 15600 0.2 0.0 0 0 ? S 08:04 0:11 [async/86]
root 15601 0.2 0.0 0 0 ? S 08:04 0:10 [async/87]
root 15602 0.2 0.0 0 0 ? S 08:04 0:12 [async/88]
root 15603 0.2 0.0 0 0 ? S 08:04 0:11 [async/89]
root 15604 0.2 0.0 0 0 ? S 08:04 0:11 [async/90]
root 15605 0.2 0.0 0 0 ? S 08:04 0:11 [async/91]
root 15606 0.2 0.0 0 0 ? S 08:04 0:11 [async/92]
root 15607 0.2 0.0 0 0 ? S 08:04 0:13 [async/93]
root 15608 0.2 0.0 0 0 ? S 08:04 0:11 [async/94]
root 15609 0.2 0.0 0 0 ? S 08:04 0:11 [async/95]
root 15610 0.2 0.0 0 0 ? S 08:04 0:11 [async/96]
root 15611 0.2 0.0 0 0 ? S 08:04 0:10 [async/97]
root 15612 0.2 0.0 0 0 ? S 08:04 0:12 [async/98]
root 15613 0.2 0.0 0 0 ? S 08:04 0:11 [async/99]
root 15614 0.2 0.0 0 0 ? S 08:04 0:11 [async/100]
root 15615 0.2 0.0 0 0 ? S 08:04 0:11 [async/101]
root 15616 0.2 0.0 0 0 ? S 08:04 0:11 [async/102]
root 15617 0.2 0.0 0 0 ? S 08:04 0:12 [async/103]
root 15618 0.2 0.0 0 0 ? S 08:04 0:11 [async/104]
root 15619 0.2 0.0 0 0 ? R 08:04 0:12 [async/105]
root 15620 0.2 0.0 0 0 ? S 08:04 0:11 [async/106]
root 15621 0.2 0.0 0 0 ? S 08:04 0:10 [async/107]
root 15622 0.2 0.0 0 0 ? S 08:04 0:12 [async/108]
root 15623 0.2 0.0 0 0 ? S 08:04 0:12 [async/109]
root 15624 0.2 0.0 0 0 ? S 08:04 0:11 [async/110]
root 15625 0.2 0.0 0 0 ? S 08:04 0:11 [async/111]
root 15626 0.2 0.0 0 0 ? S 08:04 0:11 [async/112]
root 15627 0.2 0.0 0 0 ? S 08:04 0:12 [async/113]
root 15628 0.2 0.0 0 0 ? S 08:04 0:12 [async/114]
root 15629 0.2 0.0 0 0 ? S 08:04 0:12 [async/115]
root 15630 0.2 0.0 0 0 ? S 08:04 0:12 [async/116]
root 15631 0.2 0.0 0 0 ? S 08:04 0:10 [async/117]
root 15632 0.2 0.0 0 0 ? S 08:04 0:12 [async/118]
root 15633 0.2 0.0 0 0 ? S 08:04 0:11 [async/119]
root 15634 0.2 0.0 0 0 ? S 08:04 0:11 [async/120]
root 15635 0.2 0.0 0 0 ? S 08:04 0:11 [async/121]
root 15636 0.2 0.0 0 0 ? S 08:04 0:11 [async/122]
root 15637 0.2 0.0 0 0 ? S 08:04 0:12 [async/123]
root 15638 0.2 0.0 0 0 ? R 08:04 0:11 [async/124]
root 15639 0.2 0.0 0 0 ? S 08:04 0:11 [async/125]
root 15640 0.2 0.0 0 0 ? S 08:04 0:12 [async/126]
root 15641 0.2 0.0 0 0 ? S 08:04 0:10 [async/127]
root 15642 0.2 0.0 0 0 ? S 08:04 0:12 [async/128]
root 15643 0.2 0.0 0 0 ? S 08:04 0:12 [async/129]
root 15644 0.2 0.0 0 0 ? S 08:04 0:11 [async/130]
root 15645 0.2 0.0 0 0 ? S 08:04 0:11 [async/131]
root 15646 0.2 0.0 0 0 ? S 08:04 0:11 [async/132]
root 15647 0.2 0.0 0 0 ? S 08:04 0:12 [async/133]
root 15648 0.2 0.0 0 0 ? S 08:04 0:12 [async/134]
root 15649 0.2 0.0 0 0 ? S 08:04 0:11 [async/135]
root 15650 0.2 0.0 0 0 ? S 08:04 0:11 [async/136]
root 15651 0.2 0.0 0 0 ? S 08:04 0:11 [async/137]
root 15652 0.2 0.0 0 0 ? S 08:04 0:12 [async/138]
root 15653 0.2 0.0 0 0 ? S 08:04 0:11 [async/139]
root 15654 0.2 0.0 0 0 ? S 08:04 0:11 [async/140]
root 15655 0.2 0.0 0 0 ? S 08:04 0:11 [async/141]
root 15656 0.2 0.0 0 0 ? S 08:04 0:11 [async/142]
root 15657 0.2 0.0 0 0 ? S 08:04 0:12 [async/143]
root 15658 0.2 0.0 0 0 ? S 08:04 0:11 [async/144]
root 15659 0.2 0.0 0 0 ? S 08:04 0:11 [async/145]
root 15660 0.2 0.0 0 0 ? S 08:04 0:11 [async/146]
root 15661 0.2 0.0 0 0 ? S 08:04 0:11 [async/147]
root 15662 0.2 0.0 0 0 ? S 08:04 0:12 [async/148]
root 15663 0.2 0.0 0 0 ? S 08:04 0:11 [async/149]
root 15664 0.2 0.0 0 0 ? S 08:04 0:11 [async/150]
root 15665 0.2 0.0 0 0 ? S 08:04 0:11 [async/151]
root 15666 0.2 0.0 0 0 ? S 08:04 0:11 [async/152]
root 15667 0.2 0.0 0 0 ? S 08:04 0:12 [async/153]
root 15668 0.2 0.0 0 0 ? S 08:04 0:11 [async/154]
root 15669 0.2 0.0 0 0 ? S 08:04 0:11 [async/155]
root 15670 0.2 0.0 0 0 ? S 08:04 0:11 [async/156]
root 15671 0.2 0.0 0 0 ? S 08:04 0:11 [async/157]
root 15672 0.2 0.0 0 0 ? S 08:04 0:13 [async/158]
root 15673 0.2 0.0 0 0 ? S 08:04 0:11 [async/159]
root 15674 0.2 0.0 0 0 ? S 08:04 0:11 [async/160]
root 15675 0.2 0.0 0 0 ? S 08:04 0:11 [async/161]
root 15676 0.2 0.0 0 0 ? S 08:04 0:10 [async/162]
root 15677 0.2 0.0 0 0 ? S 08:04 0:12 [async/163]
root 15678 0.2 0.0 0 0 ? S 08:04 0:11 [async/164]
root 15679 0.2 0.0 0 0 ? S 08:04 0:11 [async/165]
root 15680 0.2 0.0 0 0 ? S 08:04 0:11 [async/166]
root 15681 0.2 0.0 0 0 ? S 08:04 0:11 [async/167]
root 15682 0.2 0.0 0 0 ? S 08:04 0:12 [async/168]
root 15683 0.2 0.0 0 0 ? S 08:04 0:11 [async/169]
root 15684 0.2 0.0 0 0 ? R 08:04 0:11 [async/170]
root 15685 0.2 0.0 0 0 ? S 08:04 0:11 [async/171]
root 15686 0.2 0.0 0 0 ? S 08:04 0:10 [async/172]
root 15687 0.2 0.0 0 0 ? S 08:04 0:12 [async/173]
root 15688 0.2 0.0 0 0 ? S 08:04 0:11 [async/174]
root 15689 0.2 0.0 0 0 ? S 08:04 0:11 [async/175]
root 15690 0.2 0.0 0 0 ? S 08:04 0:11 [async/176]
root 15691 0.2 0.0 0 0 ? S 08:04 0:11 [async/177]
root 15692 0.2 0.0 0 0 ? S 08:04 0:12 [async/178]
root 15693 0.2 0.0 0 0 ? S 08:04 0:10 [async/179]
root 15694 0.2 0.0 0 0 ? S 08:04 0:12 [async/180]
root 15695 0.2 0.0 0 0 ? S 08:04 0:11 [async/181]
root 15696 0.2 0.0 0 0 ? S 08:04 0:10 [async/182]
root 15697 0.2 0.0 0 0 ? S 08:04 0:12 [async/183]
root 15698 0.2 0.0 0 0 ? R 08:04 0:11 [async/184]
root 15699 0.2 0.0 0 0 ? S 08:04 0:11 [async/185]
root 15700 0.2 0.0 0 0 ? S 08:04 0:11 [async/186]
root 15701 0.2 0.0 0 0 ? S 08:04 0:10 [async/187]
root 15702 0.2 0.0 0 0 ? R 08:04 0:12 [async/188]
root 15703 0.2 0.0 0 0 ? S 08:04 0:10 [async/189]
root 15704 0.2 0.0 0 0 ? R 08:04 0:11 [async/190]
root 15705 0.2 0.0 0 0 ? S 08:04 0:11 [async/191]
root 15706 0.2 0.0 0 0 ? R 08:04 0:11 [async/192]
root 15707 0.2 0.0 0 0 ? S 08:04 0:11 [async/193]
root 15708 0.2 0.0 0 0 ? R 08:04 0:11 [async/194]
root 15709 0.2 0.0 0 0 ? S 08:04 0:11 [async/195]
root 15710 0.2 0.0 0 0 ? S 08:04 0:11 [async/196]
root 15711 0.2 0.0 0 0 ? R 08:04 0:10 [async/197]
root 15712 0.2 0.0 0 0 ? R 08:04 0:12 [async/198]
root 15713 0.2 0.0 0 0 ? S 08:04 0:11 [async/199]
root 15714 0.2 0.0 0 0 ? S 08:04 0:11 [async/200]
root 15715 0.2 0.0 0 0 ? R 08:04 0:11 [async/201]
root 15716 0.2 0.0 0 0 ? S 08:04 0:11 [async/202]
root 15717 0.2 0.0 0 0 ? R 08:04 0:12 [async/203]
root 15718 0.2 0.0 0 0 ? R 08:04 0:11 [async/204]
root 15719 0.2 0.0 0 0 ? S 08:04 0:11 [async/205]
root 15720 0.2 0.0 0 0 ? R 08:04 0:11 [async/206]
root 15721 0.2 0.0 0 0 ? R 08:04 0:11 [async/207]
root 15722 0.2 0.0 0 0 ? S 08:04 0:12 [async/208]
root 15723 0.2 0.0 0 0 ? R 08:04 0:11 [async/209]
root 15724 0.2 0.0 0 0 ? S 08:04 0:11 [async/210]
root 15725 0.2 0.0 0 0 ? R 08:04 0:11 [async/211]
root 15726 0.2 0.0 0 0 ? S 08:04 0:11 [async/212]
root 15727 0.2 0.0 0 0 ? R 08:04 0:12 [async/213]
root 15728 0.2 0.0 0 0 ? S 08:04 0:11 [async/214]
root 15729 0.2 0.0 0 0 ? R 08:04 0:12 [async/215]
root 15730 0.2 0.0 0 0 ? S 08:04 0:11 [async/216]
root 15731 0.2 0.0 0 0 ? R 08:04 0:11 [async/217]
root 15732 0.2 0.0 0 0 ? S 08:04 0:12 [async/218]
root 15733 0.2 0.0 0 0 ? R 08:04 0:11 [async/219]
root 15734 0.2 0.0 0 0 ? S 08:04 0:11 [async/220]
root 15735 0.2 0.0 0 0 ? R 08:04 0:11 [async/221]
root 15736 0.2 0.0 0 0 ? R 08:04 0:11 [async/222]
root 15737 0.2 0.0 0 0 ? R 08:04 0:12 [async/223]
root 15738 0.2 0.0 0 0 ? S 08:04 0:11 [async/224]
root 15739 0.2 0.0 0 0 ? R 08:04 0:11 [async/225]
root 15740 0.2 0.0 0 0 ? R 08:04 0:12 [async/226]
root 15741 0.2 0.0 0 0 ? S 08:04 0:11 [async/227]
root 15742 0.2 0.0 0 0 ? R 08:04 0:12 [async/228]
root 15743 0.2 0.0 0 0 ? S 08:04 0:11 [async/229]
root 15744 0.2 0.0 0 0 ? R 08:04 0:11 [async/230]
root 15745 0.2 0.0 0 0 ? R 08:04 0:11 [async/231]
root 15746 0.2 0.0 0 0 ? S 08:04 0:11 [async/232]
root 15747 0.2 0.0 0 0 ? S 08:04 0:12 [async/233]
root 15748 0.2 0.0 0 0 ? S 08:04 0:11 [async/234]
root 15749 0.2 0.0 0 0 ? S 08:04 0:12 [async/235]
root 15750 0.2 0.0 0 0 ? S 08:04 0:12 [async/236]
root 15751 0.2 0.0 0 0 ? S 08:04 0:11 [async/237]
root 15752 0.2 0.0 0 0 ? S 08:04 0:13 [async/238]
root 15753 0.2 0.0 0 0 ? S 08:04 0:12 [async/239]
root 15754 0.2 0.0 0 0 ? R 08:04 0:12 [async/240]
root 15755 0.2 0.0 0 0 ? S 08:04 0:13 [async/241]
root 15756 0.3 0.0 0 0 ? R 08:04 0:15 [async/242]
root 15757 0.2 0.0 0 0 ? R 08:04 0:13 [async/243]
root 15758 0.2 0.0 0 0 ? R 08:04 0:13 [async/244]
root 15759 0.2 0.0 0 0 ? R 08:04 0:13 [async/245]
root 15760 0.3 0.0 0 0 ? R 08:04 0:14 [async/246]
root 15761 0.5 0.0 0 0 ? S 08:04 0:24 [async/247]
root 15762 0.3 0.0 0 0 ? R 08:04 0:16 [async/248]
root 15763 0.4 0.0 0 0 ? R 08:04 0:18 [async/249]
root 15764 0.4 0.0 0 0 ? R 08:04 0:20 [async/250]
root 15765 0.5 0.0 0 0 ? R 08:04 0:25 [async/251]
root 15766 0.8 0.0 0 0 ? R 08:04 0:37 [async/252]
root 15767 1.3 0.0 0 0 ? R 08:04 1:01 [async/253]
root 15768 1.9 0.0 0 0 ? S 08:04 1:30 [async/254]
root 15769 2.1 0.0 0 0 ? R 08:04 1:40 [async/255]
On Thu, 11 Feb 2010, Justin Piszcz wrote:
> The errors in the e-mail had bad formatting, please see the following:
> http://home.comcast.net/~jpiszcz/20100211/crash.txt
>
> In addition, the linux-raid list had a typo in it, fixed.
>
> On Thu, 11 Feb 2010, Justin Piszcz wrote:
>
>> Hello,
>>
>> While tarring and compressing (bzip2) a lot of files, the following error
>> occurred, note the output is not clean because this was taken from
>> netconsole.
>>
>> When this occurs, the host cannot be rebooted with reboot/proceses cannot
>> be killed and the box locks up. There are no apparent hardware issues.
>>
>> Before, asterisk would trigger this bug, since asterisk no longer runs on
>> this host, it ran for ~2-3 months without any problems, until now.
>>
>> Please cc me as I am not on the lists, thanks.
>>
>> Is this a md raid issue or XFS? From the trace it appears to be an XFS bug?
>>
>> Feb 11 07:38:14 l1 [ 20.270270] e100: eth1 NIC Link is Up 100 Mbps Full
>> Duplex Feb 11 07:47:54 l1 [ 600.432165] INFO: task scp:4871 blocked for
>> more than 120 seconds. Feb 11 07:47:54 l1 [ 600.432177] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11
>> 07:47:54 l1 [ 600.432182] scp D
>> Feb 11 07:47:54 l1 ffff8801eee6f14c Feb 11 07:47:54 l1 0 4871 4870
>> 0x00000000 Feb 11 07:47:54 l1 [ 600.432188] ffff880220c87950
>> Feb 11 07:47:54 l1 0000000000000082
>> Feb 11 07:47:54 l1 0000000000200200
>> Feb 11 07:47:54 l1 ffff88023eb71400
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432196] 000000000000c928
>> Feb 11 07:47:54 l1 ffff8801f7a55fd8
>> Feb 11 07:47:54 l1 ffff880205fc6150
>> Feb 11 07:47:54 l1 ffff880205fc63c8
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432203] 0000000000001000
>> Feb 11 07:47:54 l1 ffffffff8108d02a
>> Feb 11 07:47:54 l1 ffff8801eee6f0c0
>> Feb 11 07:47:54 l1 ffff880205fc63c8
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432209] Call Trace: Feb 11
>> 07:47:54 l1 [ 600.432230] [<ffffffff8108d02a>] ?
>> generic_file_buffered_write+0x1aa/0x290 Feb 11 07:47:54 l1 [ 600.432236]
>> [<ffffffff8145f02d>] ? __down_write_nested+0x7d/0xb0 Feb 11 07:47:54 l1 [
>> 600.432242] [<ffffffff8120387d>] ? xfs_write+0x23d/0x950 Feb 11 07:47:54
>> l1 [ 600.432250] [<ffffffff810ba913>] ? do_sync_write+0xe3/0x130 Feb 11
>> 07:47:54 l1 [ 600.432260] [<ffffffff8106dd90>] ?
>> autoremove_wake_function+0x0/0x30 Feb 11 07:47:54 l1 [ 600.432266]
>> [<ffffffff810e9654>] ? fsnotify+0x4/0x1a0 Feb 11 07:47:54 l1 [ 600.432271]
>> [<ffffffff8102f78e>] ? common_interrupt+0xe/0x13 Feb 11 07:47:54 l1 [
>> 600.432276] [<ffffffff810bb45b>] ? vfs_write+0xcb/0x180 Feb 11 07:47:54 l1
>> [ 600.432280] [<ffffffff810bb613>] ? sys_write+0x53/0xa0 Feb 11 07:47:54
>> l1 [ 600.432285] [<ffffffff8102eeeb>] ? system_call_fastpath+0x16/0x1b Feb
>> 11 07:47:54 l1 [ 600.432291] INFO: task flush-9:3:4874 blocked for more
>> than 120 seconds. Feb 11 07:47:54 l1 [ 600.432294] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11
>> 07:47:54 l1 [ 600.432297] flush-9:3 D
>> Feb 11 07:47:54 l1 ffff88021948d750 Feb 11 07:47:54 l1 0 4874 2
>> 0x00000000 Feb 11 07:47:54 l1 [ 600.432309] ffff88023fa3c8c0
>> Feb 11 07:47:54 l1 0000000000000046
>> Feb 11 07:47:54 l1 000000001948d750
>> Feb 11 07:47:54 l1 ffffffff8163d788
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432323] 000000000000c928
>> Feb 11 07:47:54 l1 ffff8801f79fbfd8
>> Feb 11 07:47:54 l1 ffff88021948d750
>> Feb 11 07:47:54 l1 ffff88021948d9c8
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432329] 000000013f467bb0
>> Feb 11 07:47:54 l1 ffffffff8108c183
>> Feb 11 07:47:54 l1 0000000000000000
>> Feb 11 07:47:54 l1 ffff88021948d9c8
>> Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [ 600.432337] Call Trace: Feb 11
>> 07:47:54 l1 [ 600.432341] [<ffffffff8108c183>] ? find_lock_page+0x23/0x80
>> Feb 11 07:47:54 l1 [ 600.432344] [<ffffffff8108cbb1>] ?
>> find_or_create_page+0x41/0xc0 Feb 11 07:47:54 l1 [ 600.432349]
>> [<ffffffff8145dbc5>] ? schedule_timeout+0x195/0x1f0 Feb 11 07:47:54 l1 [
>> 600.432352] [<ffffffff8145ea81>] ? __down+0x61/0xa0 Feb 11 07:47:54 l1 [
>> 600.432356] [<ffffffff81072046>] ? down+0x46/0x50 Feb 11 07:47:54 l1 [
>> 600.432360] [<ffffffff811ff4a4>] ? _xfs_buf_find+0x134/0x220 Feb 11
>> 07:47:54 l1 [ 600.432363] [<ffffffff811ff5fe>] ?
>> xfs_buf_get_flags+0x6e/0x190 Feb 11 07:47:54 l1 [ 600.432366]
>> [<ffffffff811ff732>] ? xfs_buf_read_flags+0x12/0xa0 Feb 11 07:47:54 l1 [
>> 600.432371] [<ffffffff811f5226>] ? xfs_trans_read_buf+0x1f6/0x350 Feb 11
>> 07:47:54 l1 [ 600.432377] [<ffffffff811b0268>] ? xfs_read_agf+0x68/0x190
>> Feb 11 07:47:54 l1 [ 600.432380] [<ffffffff811b03c0>] ?
>> xfs_alloc_read_agf+0x30/0xd0 Feb 11 07:47:54 l1 [ 600.432383]
>> [<ffffffff811b2519>] ? xfs_alloc_fix_freelist+0x379/0x450 Feb 11 07:47:54
>> l1 [ 600.432388] [<ffffffff811ddfe5>] ? xfs_iext_remove+0x35/0x80 Feb 11
>> 07:47:54 l1 [ 600.432393] [<ffffffff811bd83f>] ?
>> xfs_bmap_add_extent_delay_real+0x5ef/0x11a0 Feb 11 07:47:54 l1 [
>> 600.432396] [<ffffffff811f4e73>] ? xfs_trans_log_buf+0x63/0xa0 Feb 11
>> 07:47:54 l1 [ 600.432401] [<ffffffff811e7410>] ?
>> xlog_state_get_iclog_space+0x60/0x2c0 Feb 11 07:47:54 l1 [ 600.432404]
>> [<ffffffff8145f087>] ? __down_read+0x17/0xae Feb 11 07:47:54 l1 [
>> 600.432408] [<ffffffff811b29c0>] ? xfs_alloc_vextent+0x310/0x4b0 Feb 11
>> 07:47:54 l1 [ 600.432412] [<ffffffff811bf148>] ?
>> xfs_bmap_btalloc+0x598/0xa40 Feb 11 07:47:54 l1 [ 600.432417]
>> [<ffffffff811c19a2>] ? xfs_bmapi+0x9e2/0x11a0 Feb 11 07:47:54 l1 [
>> 600.432422] [<ffffffff811f3d4f>] ? xfs_trans_reserve+0x9f/0x210 Feb 11
>> 07:47:54 l1 [ 600.432425] [<ffffffff811e2efe>] ?
>> xfs_iomap_write_allocate+0x23e/0x3b0 Feb 11 07:47:54 l1 [ 600.432429]
>> [<ffffffff811e3aa0>] ? xfs_iomap+0x2c0/0x300 Feb 11 07:47:54 l1 [
>> 600.432433] [<ffffffff811fb875>] ? xfs_map_blocks+0x25/0x30 Feb 11
>> 07:47:54 l1 [ 600.432437] [<ffffffff811fc9a4>] ?
>> xfs_page_state_convert+0x414/0x6c0 Feb 11 07:47:54 l1 [ 600.432443]
>> [<ffffffff81224463>] ? radix_tree_gang_lookup_tag_slot+0xc3/0xf0 Feb 11
>> 07:47:54 l1 [ 600.432447] [<ffffffff811fcf57>] ?
>> xfs_vm_writepage+0x77/0x130 Feb 11 07:47:54 l1 [ 600.432453]
>> [<ffffffff8109249a>] ? __writepage+0xa/0x40 Feb 11 07:47:54 l1 [
>> 600.432456] [<ffffffff81092c6f>] ? write_cache_pages+0x1df/0x3c0 Feb 11
>> 07:47:54 l1 [ 600.432459] [<ffffffff81092490>] ? __writepage+0x0/0x40 Feb
>> 11 07:47:54 l1 [ 600.432464] [<ffffffff810d9092>] ?
>> writeback_single_inode+0xd2/0x390 Feb 11 07:47:54 l1 [ 600.432468]
>> [<ffffffff810d9f3f>] ? writeback_inodes_wb+0x3ff/0x5e0 Feb 11 07:47:54 l1 [
>> 600.432473] [<ffffffff810da23e>] ? wb_writeback+0x11e/0x1f0 Feb 11
>> 07:47:54 l1 [ 600.432479] [<ffffffff810630ee>] ?
>> try_to_del_timer_sync+0x5e/0x90 Feb 11 07:47:54 l1 [ 600.432484]
>> [<ffffffff810da56b>] ? wb_do_writeback+0x17b/0x180 Feb 11 07:47:54 l1 [
>> 600.432487] [<ffffffff810da5cd>] ? bdi_writeback_task+0x5d/0xa0 Feb 11
>> 07:47:54 l1 [ 600.432492] [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb
>> 11 07:47:54 l1 [ 600.432496] [<ffffffff8109e97e>] ?
>> bdi_start_fn+0x7e/0xf0 Feb 11 07:47:54 l1 [ 600.432499]
>> [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 11 07:47:54 l1 [
>> 600.432503] [<ffffffff8106dab6>] ? kthread+0x96/0xa0 Feb 11 07:47:54 l1 [
>> 600.432508] [<ffffffff8102fdfa>] ? child_rip+0xa/0x20 Feb 11 07:47:54 l1 [
>> 600.432513] [<ffffffff8106da20>] ? kthread+0x0/0xa0 Feb 11 07:47:54 l1 [
>> 600.432517] [<ffffffff8102fdf0>] ? child_rip+0x0/0x20 Feb 11 07:49:54 l1 [
>> 720.432171] INFO: task scp:4871 blocked for more than 120 seconds. Feb 11
>> 07:49:54 l1 [ 720.432187] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11
>> 07:49:54 l1 [ 720.432192] scp D
>> Feb 11 07:49:54 l1 ffff8801eee6f14c Feb 11 07:49:54 l1 0 4871 4870
>> 0x00000000 Feb 11 07:49:54 l1 [ 720.432198] ffff880220c87950
>> Feb 11 07:49:54 l1 0000000000000082
>> Feb 11 07:49:54 l1 0000000000200200
>> Feb 11 07:49:54 l1 ffff88023eb71400
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432206] 000000000000c928
>> Feb 11 07:49:54 l1 ffff8801f7a55fd8
>> Feb 11 07:49:54 l1 ffff880205fc6150
>> Feb 11 07:49:54 l1 ffff880205fc63c8
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432217] 0000000000001000
>> Feb 11 07:49:54 l1 ffffffff8108d02a
>> Feb 11 07:49:54 l1 ffff8801eee6f0c0
>> Feb 11 07:49:54 l1 ffff880205fc63c8
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432224] Call Trace: Feb 11
>> 07:49:54 l1 [ 720.432238] [<ffffffff8108d02a>] ?
>> generic_file_buffered_write+0x1aa/0x290 Feb 11 07:49:54 l1 [ 720.432243]
>> [<ffffffff8145f02d>] ? __down_write_nested+0x7d/0xb0 Feb 11 07:49:54 l1 [
>> 720.432249] [<ffffffff8120387d>] ? xfs_write+0x23d/0x950 Feb 11 07:49:54
>> l1 [ 720.432254] [<ffffffff810ba913>] ? do_sync_write+0xe3/0x130 Feb 11
>> 07:49:54 l1 [ 720.432259] [<ffffffff8106dd90>] ?
>> autoremove_wake_function+0x0/0x30 Feb 11 07:49:54 l1 [ 720.432265]
>> [<ffffffff810e9654>] ? fsnotify+0x4/0x1a0 Feb 11 07:49:54 l1 [ 720.432270]
>> [<ffffffff8102f78e>] ? common_interrupt+0xe/0x13 Feb 11 07:49:54 l1 [
>> 720.432275] [<ffffffff810bb45b>] ? vfs_write+0xcb/0x180 Feb 11 07:49:54 l1
>> [ 720.432280] [<ffffffff810bb613>] ? sys_write+0x53/0xa0 Feb 11 07:49:54
>> l1 [ 720.432285] [<ffffffff8102eeeb>] ? system_call_fastpath+0x16/0x1b Feb
>> 11 07:49:54 l1 [ 720.432291] INFO: task flush-9:3:4874 blocked for more
>> than 120 seconds. Feb 11 07:49:54 l1 [ 720.432295] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11
>> 07:49:54 l1 [ 720.432298] flush-9:3 D
>> Feb 11 07:49:54 l1 ffff88021948d750 Feb 11 07:49:54 l1 0 4874 2
>> 0x00000000 Feb 11 07:49:54 l1 [ 720.432306] ffff88023fa3c8c0
>> Feb 11 07:49:54 l1 0000000000000046
>> Feb 11 07:49:54 l1 000000001948d750
>> Feb 11 07:49:54 l1 ffffffff8163d788
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432322] 000000000000c928
>> Feb 11 07:49:54 l1 ffff8801f79fbfd8
>> Feb 11 07:49:54 l1 ffff88021948d750
>> Feb 11 07:49:54 l1 ffff88021948d9c8
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432329] 000000013f467bb0
>> Feb 11 07:49:54 l1 ffffffff8108c183
>> Feb 11 07:49:54 l1 0000000000000000
>> Feb 11 07:49:54 l1 ffff88021948d9c8
>> Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [ 720.432336] Call Trace: Feb 11
>> 07:49:54 l1 [ 720.432340] [<ffffffff8108c183>] ? find_lock_page+0x23/0x80
>> Feb 11 07:49:54 l1 [ 720.432343] [<ffffffff8108cbb1>] ?
>> find_or_create_page+0x41/0xc0 Feb 11 07:49:54 l1 [ 720.432348]
>> [<ffffffff8145dbc5>] ? schedule_timeout+0x195/0x1f0 Feb 11 07:49:54 l1 [
>> 720.432351] [<ffffffff8145ea81>] ? __down+0x61/0xa0 Feb 11 07:49:54 l1 [
>> 720.432356] [<ffffffff81072046>] ? down+0x46/0x50 Feb 11 07:49:54 l1 [
>> 720.432361] [<ffffffff811ff4a4>] ? _xfs_buf_find+0x134/0x220 Feb 11
>> 07:49:54 l1 [ 720.432365] [<ffffffff811ff5fe>] ?
>> xfs_buf_get_flags+0x6e/0x190 Feb 11 07:49:54 l1 [ 720.432368]
>> [<ffffffff811ff732>] ? xfs_buf_read_flags+0x12/0xa0 Feb 11 07:49:54 l1 [
>> 720.432373] [<ffffffff811f5226>] ? xfs_trans_read_buf+0x1f6/0x350 Feb 11
>> 07:49:54 l1 [ 720.432378] [<ffffffff811b0268>] ? xfs_read_agf+0x68/0x190
>> Feb 11 07:49:54 l1 [ 720.432382] [<ffffffff811b03c0>] ?
>> xfs_alloc_read_agf+0x30/0xd0 Feb 11 07:49:54 l1 [ 720.432386]
>> [<ffffffff811b2519>] ? xfs_alloc_fix_freelist+0x379/0x450 Feb 11 07:49:54
>> l1 [ 720.432391] [<ffffffff811ddfe5>] ? xfs_iext_remove+0x35/0x80 Feb 11
>> 07:49:54 l1 [ 720.432396] [<ffffffff811bd83f>] ?
>> xfs_bmap_add_extent_delay_real+0x5ef/0x11a0 Feb 11 07:49:54 l1 [
>> 720.432399] [<ffffffff811f4e73>] ? xfs_trans_log_buf+0x63/0xa0 Feb 11
>> 07:49:54 l1 [ 720.432404] [<ffffffff811e7410>] ?
>> xlog_state_get_iclog_space+0x60/0x2c0 Feb 11 07:49:54 l1 [ 720.432410]
>> [<ffffffff8145f087>] ? __down_read+0x17/0xae Feb 11 07:49:54 l1 [
>> 720.432415] [<ffffffff811b29c0>] ? xfs_alloc_vextent+0x310/0x4b0 Feb 11
>> 07:49:54 l1 [ 720.432420] [<ffffffff811bf148>] ?
>> xfs_bmap_btalloc+0x598/0xa40 Feb 11 07:49:54 l1 [ 720.432426]
>> [<ffffffff811c19a2>] ? xfs_bmapi+0x9e2/0x11a0 Feb 11 07:49:54 l1 [
>> 720.432430] [<ffffffff811f3d4f>] ? xfs_trans_reserve+0x9f/0x210 Feb 11
>> 07:49:54 l1 [ 720.432434] [<ffffffff811e2efe>] ?
>> xfs_iomap_write_allocate+0x23e/0x3b0 Feb 11 07:49:54 l1 [ 720.432437]
>> [<ffffffff811e3aa0>] ? xfs_iomap+0x2c0/0x300 Feb 11 07:49:54 l1 [
>> 720.432441] [<ffffffff811fb875>] ? xfs_map_blocks+0x25/0x30 Feb 11
>> 07:49:54 l1 [ 720.432445] [<ffffffff811fc9a4>] ?
>> xfs_page_state_convert+0x414/0x6c0 Feb 11 07:49:54 l1 [ 720.432451]
>> [<ffffffff81224463>] ? radix_tree_gang_lookup_tag_slot+0xc3/0xf0 Feb 11
>> 07:49:54 l1 [ 720.432455] [<ffffffff811fcf57>] ?
>> xfs_vm_writepage+0x77/0x130 Feb 11 07:49:54 l1 [ 720.432459]
>> [<ffffffff8109249a>] ? __writepage+0xa/0x40 Feb 11 07:49:54 l1 [
>> 720.432462] [<ffffffff81092c6f>] ? write_cache_pages+0x1df/0x3c0 Feb 11
>> 07:49:54 l1 [ 720.432466] [<ffffffff81092490>] ? __writepage+0x0/0x40 Feb
>> 11 07:49:54 l1 [ 720.432472] [<ffffffff810d9092>] ?
>> writeback_single_inode+0xd2/0x390 Feb 11 07:49:54 l1 [ 720.432476]
>> [<ffffffff810d9f3f>] ? writeback_inodes_wb+0x3ff/0x5e0 Feb 11 07:49:54 l1 [
>> 720.432480] [<ffffffff810da23e>] ? wb_writeback+0x11e/0x1f0 Feb 11
>> 07:49:54 l1 [ 720.432485] [<ffffffff810630ee>] ?
>> try_to_del_timer_sync+0x5e/0x90 Feb 11 07:49:54 l1 [ 720.432489]
>> [<ffffffff810da56b>] ? wb_do_writeback+0x17b/0x180 Feb 11 07:49:54 l1 [
>> 720.432493] [<ffffffff810da5cd>] ? bdi_writeback_task+0x5d/0xa0 Feb 11
>> 07:49:54 l1 [ 720.432497] [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb
>> 11 07:49:54 l1 [ 720.432500] [<ffffffff8109e97e>] ?
>> bdi_start_fn+0x7e/0xf0 Feb 11 07:49:54 l1 [ 720.432503]
>> [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 11 07:49:54 l1 [
>> 720.432506] [<ffffffff8106dab6>] ? kthread+0x96/0xa0 Feb 11 07:49:54 l1 [
>> 720.432511] [<ffffffff8102fdfa>] ? child_rip+0xa/0x20 Feb 11 07:49:54 l1 [
>> 720.432515] [<ffffffff8106da20>] ? kthread+0x0/0xa0 Feb 11 07:49:54 l1 [
>> 720.432519] [<ffffffff8102fdf0>] ? child_rip+0x0/0x20
>>
>> Justin.
>>
>>
>
^ permalink raw reply [flat|nested] 4+ messages in thread