From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andi Kleen Subject: Re: bad TSO performance in 2.6.9-rc2-BK Date: Thu, 30 Sep 2004 11:29:26 +0200 Sender: netdev-bounce@oss.sgi.com Message-ID: <20040930092926.GA18353@wotan.suse.de> References: <415B24C0.2020208@us.ibm.com> <20040929145050.71afa1ac.davem@davemloft.net> <20040929215613.GC26714@wotan.suse.de> <20040929162923.796d142e.davem@davemloft.net> <20040930000515.GA10496@gondor.apana.org.au> <20040929213310.40f5f33a.davem@davemloft.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Herbert Xu , ak@suse.de, niv@us.ibm.com, jheffner@psc.edu, andy.grover@gmail.com, anton@samba.org, netdev@oss.sgi.com Return-path: To: "David S. Miller" Content-Disposition: inline In-Reply-To: <20040929213310.40f5f33a.davem@davemloft.net> Errors-to: netdev-bounce@oss.sgi.com List-Id: netdev.vger.kernel.org > Right. This patch below combines that with adjustment of socket > send queue usage when we trim the head. I tested this patch on rc3. First it doesn't crash anymore. Thanks. Talking to another rc3 kernel with tg3 tso on is as fast as tso off now. Unfortunately the ACKs still look funny. Talking to a 2.6.5 kernel tso on is still about 20 MB/s slower than the non TSO case, so the problem seems to be still there. -Andi 2.6.9rc3/e1000/CSA/TSO -> 2.6.9rc3/tg3/33MhzPCI Fast. Still stretch ack. 11:13:25.721704 10.23.202.15.32777 > 10.23.202.31.32784: . ack 7241 win 5068 (DF) 11:13:25.721709 10.23.202.15.32777 > 10.23.202.31.32784: . ack 8689 win 5792 (DF) 11:13:25.721715 10.23.202.15.32777 > 10.23.202.31.32784: . ack 10137 win 6516 (DF) 11:13:25.721726 10.23.202.15.32777 > 10.23.202.31.32784: . ack 11585 win 7240 (DF) 11:13:25.721731 10.23.202.15.32777 > 10.23.202.31.32784: . ack 13033 win 7964 (DF) 11:13:25.721882 10.23.202.31.32784 > 10.23.202.15.32777: . 13033:14481(1448) ack 1 win 1460 (DF) 11:13:25.721887 10.23.202.31.32784 > 10.23.202.15.32777: . 14481:15929(1448) ack 1 win 1460 (DF) 11:13:25.721893 10.23.202.31.32784 > 10.23.202.15.32777: P 15929:16385(456) ack 1 win 1460 (DF) 11:13:25.721940 10.23.202.31.32784 > 10.23.202.15.32777: . 16385:17833(1448) ack 1 win 1460 (DF) 11:13:25.721943 10.23.202.31.32784 > 10.23.202.15.32777: . 17833:19281(1448) ack 1 win 1460 (DF) 11:13:25.721947 10.23.202.31.32784 > 10.23.202.15.32777: . 19281:20729(1448) ack 1 win 1460 (DF) 11:13:25.721951 10.23.202.31.32784 > 10.23.202.15.32777: . 20729:22177(1448) ack 1 win 1460 (DF) 11:13:25.721970 10.23.202.15.32777 > 10.23.202.31.32784: . ack 14481 win 8688 (DF) 11:13:25.721975 10.23.202.15.32777 > 10.23.202.31.32784: . ack 15929 win 9412 (DF) 11:13:25.721980 10.23.202.15.32777 > 10.23.202.31.32784: . ack 16385 win 9412 (DF) 11:13:25.721985 10.23.202.15.32777 > 10.23.202.31.32784: . ack 17833 win 10136 (DF) 11:13:25.721991 10.23.202.15.32777 > 10.23.202.31.32784: . ack 19281 win 10860 (DF) 11:13:25.721996 10.23.202.15.32777 > 10.23.202.31.32784: . ack 20729 win 11584 (DF) 11:13:25.722018 10.23.202.15.32777 > 10.23.202.31.32784: . ack 22177 win 12308 (DF) 11:13:25.722127 10.23.202.31.32784 > 10.23.202.15.32777: . 22177:23625(1448) ack 1 win 1460 (DF) 11:13:25.722134 10.23.202.15.32777 > 10.23.202.31.32784: . ack 23625 win 13032 (DF) 11:13:25.722139 10.23.202.31.32784 > 10.23.202.15.32777: . 23625:25073(1448) ack 1 win 1460 (DF) 11:13:25.722142 10.23.202.15.32777 > 10.23.202.31.32784: . ack 25073 win 13756 (DF) 11:13:25.722147 10.23.202.31.32784 > 10.23.202.15.32777: . 25073:26521(1448) ack 1 win 1460 (DF) 11:13:25.722151 10.23.202.15.32777 > 10.23.202.31.32784: . ack 26521 win 14480 (DF) 11:13:25.722156 10.23.202.31.32784 > 10.23.202.15.32777: . 26521:27969(1448) ack 1 win 1460 (DF) 11:13:25.722160 10.23.202.15.32777 > 10.23.202.31.32784: . ack 27969 win 14962 (DF) 11:13:25.722228 10.23.202.31.32784 > 10.23.202.15.32777: . 27969:29417(1448) ack 1 win 1460 (DF) 11:13:25.722236 10.23.202.31.32784 > 10.23.202.15.32777: . 29417:30865(1448) ack 1 win 1460 (DF) 11:13:25.722243 10.23.202.31.32784 > 10.23.202.15.32777: . 30865:32313(1448) ack 1 win 1460 (DF) 11:13:25.722249 10.23.202.31.32784 > 10.23.202.15.32777: . 32313:33761(1448) ack 1 win 1460 (DF) 11:13:25.722273 10.23.202.31.32784 > 10.23.202.15.32777: P 33761:35209(1448) ack 1 win 1460 (DF) 11:13:25.722277 10.23.202.31.32784 > 10.23.202.15.32777: . 35209:36657(1448) ack 1 win 1460 (DF) 11:13:25.722282 10.23.202.31.32784 > 10.23.202.15.32777: . 36657:38105(1448) ack 1 win 1460 (DF) 11:13:25.722339 10.23.202.15.32777 > 10.23.202.31.32784: . ack 38105 win 15204 (DF) 11:13:25.722415 10.23.202.31.32784 > 10.23.202.15.32777: . 38105:39553(1448) ack 1 win 1460 (DF) 11:13:25.722419 10.23.202.31.32784 > 10.23.202.15.32777: . 39553:41001(1448) ack 1 win 1460 (DF) 11:13:25.722425 10.23.202.31.32784 > 10.23.202.15.32777: . 41001:42449(1448) ack 1 win 1460 (DF) 11:13:25.722429 10.23.202.31.32784 > 10.23.202.15.32777: . 42449:43897(1448) ack 1 win 1460 (DF) 11:13:25.722433 10.23.202.31.32784 > 10.23.202.15.32777: . 43897:45345(1448) ack 1 win 1460 (DF) 11:13:25.722436 10.23.202.31.32784 > 10.23.202.15.32777: . 45345:46793(1448) ack 1 win 1460 (DF) 11:13:25.722459 10.23.202.31.32784 > 10.23.202.15.32777: . 46793:48241(1448) ack 1 win 1460 (DF) 11:13:25.722463 10.23.202.31.32784 > 10.23.202.15.32777: . 48241:49689(1448) ack 1 win 1460 (DF) 11:13:25.722472 10.23.202.15.32777 > 10.23.202.31.32784: . ack 39553 win 15928 (DF) 11:13:25.722504 10.23.202.15.32777 > 10.23.202.31.32784: . ack 49689 win 15928 (DF) 11:13:25.722616 10.23.202.31.32784 > 10.23.202.15.32777: . 49689:51137(1448) ack 1 win 1460 (DF) 11:13:25.722622 10.23.202.31.32784 > 10.23.202.15.32777: . 51137:52585(1448) ack 1 win 1460 (DF) 11:13:25.722690 10.23.202.31.32784 > 10.23.202.15.32777: . 52585:54033(1448) ack 1 win 1460 (DF) 11:13:25.722697 10.23.202.31.32784 > 10.23.202.15.32777: . 54033:55481(1448) ack 1 win 1460 (DF) 11:13:25.722703 10.23.202.31.32784 > 10.23.202.15.32777: . 55481:56929(1448) ack 1 win 1460 (DF) 11:13:25.722710 10.23.202.31.32784 > 10.23.202.15.32777: . 56929:58377(1448) ack 1 win 1460 (DF) 11:13:25.722716 10.23.202.31.32784 > 10.23.202.15.32777: . 58377:59825(1448) ack 1 win 1460 (DF) 11:13:25.722786 10.23.202.31.32784 > 10.23.202.15.32777: P 59825:61273(1448) ack 1 win 1460 (DF) 11:13:25.722793 10.23.202.31.32784 > 10.23.202.15.32777: . 61273:62721(1448) ack 1 win 1460 (DF) 11:13:25.722798 10.23.202.31.32784 > 10.23.202.15.32777: . 62721:64169(1448) ack 1 win 1460 (DF) 2.6.9rc3/e1000/CSA/TSO -> 2.6.5/tg3/33MhzPCI Slow. Extreme stretch ack: 11:19:45.486497 10.23.202.31.32782 > 10.23.202.10.32978: . 27969:29417(1448) ack 1 win 1460 (DF) 11:19:45.486506 10.23.202.10.32978 > 10.23.202.31.32782: . ack 29417 win 62264 (DF) 11:19:45.486510 10.23.202.31.32782 > 10.23.202.10.32978: P 29417:30865(1448) ack 1 win 1460 (DF) 11:19:45.486536 10.23.202.31.32782 > 10.23.202.10.32978: . 30865:32313(1448) ack 1 win 1460 (DF) 11:19:45.486543 10.23.202.31.32782 > 10.23.202.10.32978: . 32313:33761(1448) ack 1 win 1460 (DF) 11:19:45.486642 10.23.202.10.32978 > 10.23.202.31.32782: . ack 33761 win 63712 (DF) 11:19:45.486678 10.23.202.31.32782 > 10.23.202.10.32978: . 33761:35209(1448) ack 1 win 1460 (DF) 11:19:45.486686 10.23.202.10.32978 > 10.23.202.31.32782: . ack 35209 win 63712 (DF) 11:19:45.486691 10.23.202.31.32782 > 10.23.202.10.32978: . 35209:36657(1448) ack 1 win 1460 (DF) 11:19:45.486697 10.23.202.31.32782 > 10.23.202.10.32978: . 36657:38105(1448) ack 1 win 1460 (DF) 11:19:45.486704 10.23.202.31.32782 > 10.23.202.10.32978: . 38105:39553(1448) ack 1 win 1460 (DF) 11:19:45.486710 10.23.202.31.32782 > 10.23.202.10.32978: . 39553:41001(1448) ack 1 win 1460 (DF) 11:19:45.486716 10.23.202.31.32782 > 10.23.202.10.32978: . 41001:42449(1448) ack 1 win 1460 (DF) 11:19:45.486722 10.23.202.31.32782 > 10.23.202.10.32978: . 42449:43897(1448) ack 1 win 1460 (DF) 11:19:45.486747 10.23.202.31.32782 > 10.23.202.10.32978: . 43897:45345(1448) ack 1 win 1460 (DF) 11:19:45.486754 10.23.202.31.32782 > 10.23.202.10.32978: . 45345:46793(1448) ack 1 win 1460 (DF) 11:19:45.486761 10.23.202.31.32782 > 10.23.202.10.32978: . 46793:48241(1448) ack 1 win 1460 (DF) 11:19:45.486768 10.23.202.31.32782 > 10.23.202.10.32978: . 48241:49689(1448) ack 1 win 1460 (DF) 11:19:45.486775 10.23.202.31.32782 > 10.23.202.10.32978: . 49689:51137(1448) ack 1 win 1460 (DF) 11:19:45.486859 10.23.202.31.32782 > 10.23.202.10.32978: . 51137:52585(1448) ack 1 win 1460 (DF) 11:19:45.486867 10.23.202.31.32782 > 10.23.202.10.32978: . 52585:54033(1448) ack 1 win 1460 (DF) 11:19:45.486873 10.23.202.31.32782 > 10.23.202.10.32978: . 54033:55481(1448) ack 1 win 1460 (DF) 11:19:45.486880 10.23.202.31.32782 > 10.23.202.10.32978: P 55481:56929(1448) ack 1 win 1460 (DF) 11:19:45.487018 10.23.202.10.32978 > 10.23.202.31.32782: . ack 56929 win 63712 (DF) 11:19:45.487279 10.23.202.31.32782 > 10.23.202.10.32978: . 56929:58377(1448) ack 1 win 1460 (DF) 11:19:45.487288 10.23.202.31.32782 > 10.23.202.10.32978: . 58377:59825(1448) ack 1 win 1460 (DF) 11:19:45.487294 10.23.202.31.32782 > 10.23.202.10.32978: . 59825:61273(1448) ack 1 win 1460 (DF) 11:19:45.487299 10.23.202.31.32782 > 10.23.202.10.32978: . 61273:62721(1448) ack 1 win 1460 (DF) 11:19:45.487305 10.23.202.31.32782 > 10.23.202.10.32978: . 62721:64169(1448) ack 1 win 1460 (DF) 11:19:45.487311 10.23.202.31.32782 > 10.23.202.10.32978: . 64169:65617(1448) ack 1 win 1460 (DF) 11:19:45.487316 10.23.202.31.32782 > 10.23.202.10.32978: . 65617:67065(1448) ack 1 win 1460 (DF) 11:19:45.487370 10.23.202.31.32782 > 10.23.202.10.32978: . 67065:68513(1448) ack 1 win 1460 (DF) 11:19:45.487376 10.23.202.31.32782 > 10.23.202.10.32978: . 68513:69961(1448) ack 1 win 1460 (DF) 11:19:45.487382 10.23.202.31.32782 > 10.23.202.10.32978: . 69961:71409(1448) ack 1 win 1460 (DF) 11:19:45.487388 10.23.202.31.32782 > 10.23.202.10.32978: . 71409:72857(1448) ack 1 win 1460 (DF) 11:19:45.487393 10.23.202.31.32782 > 10.23.202.10.32978: . 72857:74305(1448) ack 1 win 1460 (DF) 11:19:45.487398 10.23.202.31.32782 > 10.23.202.10.32978: . 74305:75753(1448) ack 1 win 1460 (DF) 11:19:45.487421 10.23.202.31.32782 > 10.23.202.10.32978: . 75753:77201(1448) ack 1 win 1460 (DF) 11:19:45.487427 10.23.202.31.32782 > 10.23.202.10.32978: . 77201:78649(1448) ack 1 win 1460 (DF) 11:19:45.487433 10.23.202.31.32782 > 10.23.202.10.32978: . 78649:80097(1448) ack 1 win 1460 (DF) 11:19:45.487563 10.23.202.10.32978 > 10.23.202.31.32782: . ack 80097 win 63712 (DF) 2.6.9rc3/e1000/CSA/noTSO -> 2.6.5/tg3/33MhzPCI Fast. Also stretch ACK, but less extreme. 11:26:12.886397 10.23.202.10.32979 > 10.23.202.31.32788: . ack 8689 win 23168 (DF) 11:26:12.886405 10.23.202.10.32979 > 10.23.202.31.32788: . ack 10137 win 26064 (DF) 11:26:12.886420 10.23.202.10.32979 > 10.23.202.31.32788: . ack 11585 win 28960 (DF) 11:26:12.886440 10.23.202.10.32979 > 10.23.202.31.32788: . ack 13033 win 31856 (DF) 11:26:12.886488 10.23.202.31.32788 > 10.23.202.10.32979: . 13033:14481(1448) ack 1 win 1460 (DF) 11:26:12.886500 10.23.202.10.32979 > 10.23.202.31.32788: . ack 14481 win 34752 (DF) 11:26:12.886504 10.23.202.31.32788 > 10.23.202.10.32979: . 14481:15929(1448) ack 1 win 1460 (DF) 11:26:12.886513 10.23.202.10.32979 > 10.23.202.31.32788: . ack 15929 win 37648 (DF) 11:26:12.886749 10.23.202.31.32788 > 10.23.202.10.32979: . 15929:17377(1448) ack 1 win 1460 (DF) 11:26:12.886757 10.23.202.31.32788 > 10.23.202.10.32979: . 17377:18825(1448) ack 1 win 1460 (DF) 11:26:12.886763 10.23.202.31.32788 > 10.23.202.10.32979: . 18825:20273(1448) ack 1 win 1460 (DF) 11:26:12.886769 10.23.202.31.32788 > 10.23.202.10.32979: . 20273:21721(1448) ack 1 win 1460 (DF) 11:26:12.886775 10.23.202.31.32788 > 10.23.202.10.32979: . 21721:23169(1448) ack 1 win 1460 (DF) 11:26:12.886780 10.23.202.31.32788 > 10.23.202.10.32979: . 23169:24617(1448) ack 1 win 1460 (DF) 11:26:12.886786 10.23.202.31.32788 > 10.23.202.10.32979: P 24617:26065(1448) ack 1 win 1460 (DF) 11:26:12.886792 10.23.202.31.32788 > 10.23.202.10.32979: P 26065:27513(1448) ack 1 win 1460 (DF) 11:26:12.886798 10.23.202.31.32788 > 10.23.202.10.32979: . 27513:28961(1448) ack 1 win 1460 (DF) 11:26:12.886805 10.23.202.31.32788 > 10.23.202.10.32979: . 28961:30409(1448) ack 1 win 1460 (DF) 11:26:12.886810 10.23.202.31.32788 > 10.23.202.10.32979: . 30409:31857(1448) ack 1 win 1460 (DF) 11:26:12.886816 10.23.202.31.32788 > 10.23.202.10.32979: P 31857:32769(912) ack 1 win 1460 (DF) 11:26:12.886892 10.23.202.10.32979 > 10.23.202.31.32788: . ack 17377 win 40544 (DF) 11:26:12.886901 10.23.202.10.32979 > 10.23.202.31.32788: . ack 18825 win 43440 (DF) 11:26:12.886910 10.23.202.10.32979 > 10.23.202.31.32788: . ack 20273 win 46336 (DF) 11:26:12.886917 10.23.202.10.32979 > 10.23.202.31.32788: . ack 21721 win 49232 (DF) 11:26:12.886925 10.23.202.10.32979 > 10.23.202.31.32788: . ack 23169 win 52128 (DF) 11:26:12.886933 10.23.202.10.32979 > 10.23.202.31.32788: . ack 24617 win 55024 (DF) 11:26:12.886960 10.23.202.10.32979 > 10.23.202.31.32788: . ack 26065 win 57920 (DF) 11:26:12.886976 10.23.202.10.32979 > 10.23.202.31.32788: . ack 27513 win 60816 (DF) 11:26:12.886985 10.23.202.10.32979 > 10.23.202.31.32788: . ack 28961 win 63712 (DF) 11:26:12.886995 10.23.202.10.32979 > 10.23.202.31.32788: . ack 30409 win 63712 (DF) 11:26:12.887027 10.23.202.10.32979 > 10.23.202.31.32788: . ack 31857 win 63712 (DF) 11:26:12.887035 10.23.202.10.32979 > 10.23.202.31.32788: . ack 32769 win 63712 (DF) 11:26:12.887151 10.23.202.31.32788 > 10.23.202.10.32979: . 32769:34217(1448) ack 1 win 1460 (DF) 11:26:12.887161 10.23.202.31.32788 > 10.23.202.10.32979: . 34217:35665(1448) ack 1 win 1460 (DF) 11:26:12.887167 10.23.202.31.32788 > 10.23.202.10.32979: . 35665:37113(1448) ack 1 win 1460 (DF) 11:26:12.887172 10.23.202.31.32788 > 10.23.202.10.32979: . 37113:38561(1448) ack 1 win 1460 (DF) 11:26:12.887225 10.23.202.31.32788 > 10.23.202.10.32979: . 38561:40009(1448) ack 1 win 1460 (DF) 11:26:12.887231 10.23.202.31.32788 > 10.23.202.10.32979: . 40009:41457(1448) ack 1 win 1460 (DF) 11:26:12.887237 10.23.202.31.32788 > 10.23.202.10.32979: . 41457:42905(1448) ack 1 win 1460 (DF) 11:26:12.887283 10.23.202.31.32788 > 10.23.202.10.32979: . 42905:44353(1448) ack 1 win 1460 (DF) 11:26:12.887289 10.23.202.31.32788 > 10.23.202.10.32979: . 44353:45801(1448) ack 1 win 1460 (DF) 11:26:12.887343 10.23.202.31.32788 > 10.23.202.10.32979: . 45801:47249(1448) ack 1 win 1460 (DF) 11:26:12.887349 10.23.202.31.32788 > 10.23.202.10.32979: . 47249:48697(1448) ack 1 win 1460 (DF) 11:26:12.887355 10.23.202.31.32788 > 10.23.202.10.32979: . 48697:50145(1448) ack 1 win 1460 (DF) 11:26:12.887362 10.23.202.31.32788 > 10.23.202.10.32979: P 50145:51593(1448) ack 1 win 1460 (DF) 11:26:12.887369 10.23.202.31.32788 > 10.23.202.10.32979: . 51593:53041(1448) ack 1 win 1460 (DF) 11:26:12.887377 10.23.202.31.32788 > 10.23.202.10.32979: . 53041:54489(1448) ack 1 win 1460 (DF) 11:26:12.887383 10.23.202.31.32788 > 10.23.202.10.32979: . 54489:55937(1448) ack 1 win 1460 (DF) 11:26:12.887389 10.23.202.31.32788 > 10.23.202.10.32979: . 55937:57385(1448) ack 1 win 1460 (DF) 11:26:12.887413 10.23.202.31.32788 > 10.23.202.10.32979: . 57385:58833(1448) ack 1 win 1460 (DF) 11:26:12.887419 10.23.202.31.32788 > 10.23.202.10.32979: . 58833:60281(1448) ack 1 win 1460 (DF) 11:26:12.887425 10.23.202.31.32788 > 10.23.202.10.32979: . 60281:61729(1448) ack 1 win 1460 (DF) 11:26:12.887431 10.23.202.31.32788 > 10.23.202.10.32979: . 61729:63177(1448) ack 1 win 1460 (DF) 11:26:12.887622 10.23.202.10.32979 > 10.23.202.31.32788: . ack 63177 win 63712 (DF) 11:26:12.887765 10.23.202.31.32788 > 10.23.202.10.32979: . 63177:64625(1448) ack 1 win 1460 (DF) 11:26:12.887774 10.23.202.31.32788 > 10.23.202.10.32979: . 64625:66073(1448) ack 1 win 1460 (DF)